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Abstract 


Previous  research  on  completed  defense  R&D  shows  that  contract  expenditures 
can  be  fit  well  with  a  Rayleigh  model.  With  fixed  outlay  rates,  as  prescribed  by  the  OSD 
comptroller,  the  budget  profile  must  have  most  of  the  funds  in  the  early  years  to  produce 
Rayleigh-distributed  expenditures.  R&D  programs  with  more  delayed  funding  profiles 
may  also  produce  expenditures  that  a  Rayleigh  model  fits  through  schedule  slips  and  cost 
overruns.  This  research  tests  how  well  the  initial  funding  profile  produces  Rayleigh- 
distributed  expenditures  that  can  be  related  to  the  program’s  final  cost  overrun  and 
schedule  slips.  Based  only  on  the  initial  budget  profile,  we  explain  53.4%  of  cost 
overruns  and  50.5%  of  percent  schedule  slip  in  37  completed  programs. 

Research  Value 

Our  research  indicates  to  the  services  the  importance  of  budgeting  funds 
appropriately  in  the  program  to  reduce  the  frequency  of  cost  overruns  and  schedule  slips. 
The  military  services’  financial  communities,  OSD  Program  Analysis  and  Evaluation, 
and  OSD  comptroller  may  use  the  method  to  ensure  that  R&D  programs  are  adequately 
funded  sufficiently  early  to  reduce  program  cost  and  schedule  growth. 


xi 


I.  Introduction 


Problem  Statement 

Military  Research  and  Development  (R&D)  programs  have  difficulties  with  cost 
overruns  and  schedule  slips.  This  study  hypothesizes  a  correlation  between  initial  R&D 
program’s  budget  profiles  and  program  results.  Based  on  evidence  that  historical 
expenditures  follow  the  Rayleigh  model,  this  study  estimates  a  relationship  between  the 
initial  budget  profile  and  both  the  length  of  schedule  slip  and  the  amount  of  program  cost 
overrun. 

Background 

Research  and  Development  (R&D)  cost  growth  is  a  persistent  problem  for  the 
Department  of  Defense  (DoD).  As  programs  mature,  their  funding  profiles  tend  to 
change  significantly  from  the  initial  conception.  The  profile  change  indicates  that  the 
initial  budget  profile  was  inappropriate  for  the  program.  Funding  is  inappropriate  if  a 
program  has  adequate  funding,  but  it  becomes  available  in  the  wrong  years.  Moreover,  if 
initial  funding  is  inadequate  or  inappropriate  to  actual  program  funding  needs,  schedule 
slips  and  cost  overruns  may  result. 

Belcher  and  Dukovich  state  that  constrained  funding  is  one  of  the  factors  that 
causes  productivity  inefficiencies,  resulting  in  schedule  slips  and  greater  program  cost 
(Belcher,  1999).  Several  studies  support  the  notion  that  the  Rayleigh  probability 
distribution  is  appropriate  for  modeling  R&D  budget  profiles.  Watkins  (1982)  and 
Abernathy  (1984)  concluded  that  the  Raleigh  fit  R&D  program  cost  data.  Lee,  Hogue, 
and  Gallagher  (1997)  show  that  program  expenditures  fit  the  Rayleigh  distribution.  In 
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the  study,  they  devise  two  techniques  for  using  a  Rayleigh  model  to  develop  an  efficient 
funding  profile,  given  a  point  estimate  of  total  development  cost  and  desired  program 
completion  time.  They  base  their  method  on  the  concept  that  a  budget  that  supports 
expenditures  that  follow  the  Rayleigh  model  is  an  efficient  program.  Efficient  implies 
minimal  program  cost  and  schedule  increases. 

The  Rayleigh  model  is  a  function  of  time  t,  scale  d,  and  shape  parameter  a.  The 
Rayleigh  probability  density  function  is  f(t)  =  2ateat .  The  scale  parameter  d relates  to 
total  cost  of  the  program  and  the  shape  parameter  a  relates  to  the  program  completion 
time.  Figure  1-1  depicts  the  effect  of  the  shape  parameter  a,  with  d  and  t  identical  for 
each  curve. 


Figure  1-1.  Rayleigh  Probability  Density  Function 

Lee,  Hogue,  and  Gallagher  (1997)  develop  budget  profiles  that  support  Rayleigh 
distributed  expenditures.  If  the  associated  program  expenditures  follow  a  Rayleigh 
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model,  these  budgets  should  minimize  cost  overruns  and  schedule  slips.  The  focus  of  this 
research  is  to  determine  whether  R&D  budgeting  contributes  to  cost  overruns  and  delays 
by  not  having  funding  when  needed. 

Scope 

Past  research  indicates  that  budgets  supporting  Rayleigh  distributed  expenditures 
are  efficient.  We  test  the  impact  of  deviations  from  Rayleigh  expenditures  on  program 
cost  and  schedule  growth. 

Research  Approach 

The  change  in  shape  of  a  budget  profile  and  the  resulting  ending  budget  profile 
are  dependent  on  many  program  factors.  Program  risk,  program  priority,  annual  budget, 
and  program  slips  all  play  a  role  in  defining  how  a  budget  profile  develops  over  time. 

This  study,  however,  seeks  a  correlation  between  a  single  indicator,  the  initial  funding 
profile  and  the  final  expenditures.  We  approach  confirming  the  relationship  between 
initial  funding  profiles  and  Rayleigh  expenditure  profiles  in  two  separate  methods. 

The  first  method  assesses  the  cumulative  expenditure  distribution  with  a  Rayleigh 
goodness  of  fit  test.  We  evaluate  the  initial  expenditures  by  determining  whether  there  is 
sufficient  statistical  evidence  to  conclude  they  are  Rayleigh  using  Anderson-Darling, 
Cramer-Von  Mises,  and  Kolmolgorov  goodness  of  fit  statistics. 

The  second  method  determines  Weibull  parameters  from  the  initial  expenditures. 
The  Weibull  distribution,  which  is  an  extremely  flexible  distribution,  includes  the 
Rayleigh  distribution. 
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While  Abernathy  (1984)  and  Lee,  Hougue,  and  Gallagher  (1997)  showed  that 
contractor  costs  follow  the  Rayleigh  profile,  contract  costs  represent  a  subset,  albeit 
significant,  of  total  program  costs.  In  first  step,  we  examine  whether  total  program  final 
expenditures  also  follow  the  Rayleigh  profile.  This  relationship  likely  exists,  as 
contractor  costs  are  the  major  subset  of  total  program  costs.  We  use  Selective 
Acquisition  Reports  (SAR)  data,  which  provides  funding  data  for  large  systems,  to 
determine  how  well  total  program  cost  may  be  modeled  by  a  Rayleigh  distribution. 

After  establishing  the  quality  of  the  Rayleigh  model,  we  project  final  cost 
overruns  and  program  slips  from  initial  budget  profiles  for  a  variety  of  programs.  We 
determine  expected  expenditure  profiles  given  initial  budget  profiles  from  the  SAR 
database  with  Office  of  the  Secretary  of  Defense  (OSD)  Comptroller  directed  outlay 
rates.  Also,  we  compare  the  Rayleigh  expenditure  profile  to  the  expected  expenditure 
profiles.  The  differences  between  the  initial  expected  expenditure  profiles  and  the 
Rayleigh  derived  profiles  provide  the  information  to  explain  schedule  slips  and  cost 
overruns. 

Research  Benefits 

This  research  evaluates  historical  program  cost  overruns  and  schedule  slips  based 
on  the  initial  funding  profile.  Since  this  research  assesses  the  impact  of  an  initial  R&D 
budget  profile  on  program  growth,  the  results  are  potentially  useful  to  all  levels  of 
acquisition  analysts  and  managers  from  the  system  program  office  to  OSD  Program 
Analysis  and  Evaluation  (PA&E). 
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Chapter  Two  explores  previous  research  which  provides  a  foundation  for  the 
Rayleigh  theory  and  subsequent  corollaries.  Chapter  Three  explains  the  methodology  of 
evaluating  historical  DoD  program  data  for  its  capacity  to  predict  program  growth  and 
provides  the  results  of  the  data  analysis.  Chapter  Four  provides  conclusions  and 
recommendations  for  further  effort. 
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II.  Literature  Review  and  Concept  Definition 


Chapter  Overview 

Learning  curve  theory  provides  a  typical  approach  to  estimating  certain 
components  of  production  programs.  The  learning  curve  provides  a  mathematical 
relationship  between  the  cost  of  the  component  and  the  quantity  of  components  being 
produced.  Generic  learning  curve  theory  states  that  as  the  quantity  of  components 
doubles,  the  cost  per  component  decreases  by  a  constant  percentage.  Along  with 
providing  a  suitable  estimate  of  cost,  this  approach  can  also  provide  an  idea  of  when 
funding  is  required.  Unfortunately,  a  standard  analogous  approach  for  estimating 
Research  and  Development  (R&D)  programs  does  not  currently  exist. 

Establishing  an  appropriate  funding  profile  for  development  programs  is  critical 
to  their  financial  success.  Inefficiently  funded  programs  can  either  consume  funding 
needed  elsewhere,  or  cause  program  management  crises  in  the  form  of  funding  shortfalls. 
In  the  absence  of  a  standard  model  to  establish  development  program  costs,  theorists 
developed  mathematical  models  to  derive  the  appropriate  development  funding  profiles. 

This  chapter  presents  several  mathematical  models  used  to  estimate  development 
program  effort.  These  models  include  the  Beta  model,  the  Sech-Squared  Model,  and  the 
Rayleigh  model.  As  the  following  effort  is  based  on  Rayleigh  theory,  the  Rayleigh  model 
will  be  explored  in  depth.  Finally,  we  establish  a  relationship  between  the  Rayleigh 
model  and  the  Weibull  distribution. 

Each  of  the  models  provides  an  approach  for  modeling  development  program 
expenditures.  Each  of  the  models  approximates  constant  years  expenditures  by  applying 
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a  scaling  factor,  total  expenditures  in  constant  dollars,  to  the  distribution.  Therefore,  an 
initial  discussion  of  converting  budget  profiles  to  expenditures  is  necessary. 

Conversion  from  Budgets  to  Expenditures 

Multi-year  appropriations  and  inflation  affect  the  computation  of  DoD  budgets. 
DoD  must  spend  at  least  a  certain  percentage  of  the  money  allocated  for  R&D  programs 
in  a  particular  year,  as  prescribed  by  the  Office  of  the  Secretary  of  Defense  Comptroller 
OSD(C).  This  percentage  is  called  an  outlay  rate.  Inflation  also  has  a  significant  effect 
on  the  calculation  of  funding  needed  for  outyear  budgets.  The  combined  effect  of  outlay 
rates  and  inflation  implies  that  data  in  the  form  of  budget  profiles  must  be  converted  to 
expenditures  in  constant  dollars  to  match  the  models.  To  reflect  the  expenditure  amounts 
devoid  of  the  effects  of  inflation,  then  year  budget  dollars  need  to  be  converted  to 
constant  year  expenditures.  To  accomplish  this  conversion  we  use 

Oj  =  BjSi  +  Bi-iS?  +  Bj.ySs  +  ...  +  Bj.jSj  (1) 

where  O,  represents  the  expenditure  for  year  i  in  then  year  dollars,  B,  the  initial  authorized 
budget  amount,  Si,  the  percentage  outlay  rate  for  a  given  year  in  the  outlay  profile,  and  J 
the  number  of  years  in  the  outlay  profile.  Since  the  intent  is  to  outlay  the  entire  budgeted 
amount, 

IX  ^ 

Outlay  rates  may  not  sum  to  one,  since  DoD  historically  does  not  outlay  its  entire  budget. 
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Since  the  outlays  in  (1)  are  still  in  then  year  dollars,  they  must  be  converted  to 
constant  year  dollars  to  be  properly  used  in  the  expenditure  models.  To  remove  inflation 
from  each  0„  the  annual  expenditure  must  de  deflated.  Given  the  appropriate  inflation 
factor, 

O*  =  Oj/  Cj  (2) 

* 

where  Oj  is  the  expenditure  for  z'th  year  in  base-year  dollars,  c,  is  the  inflation  factor  of 
the  /th  year,  and  O,  is  the  expenditures  for  z'th  year  in  then-year  dollars.  The  constant- 
year  dollar  expenditures  are  necessary  for  the  models  since  they  theoretically  account  for 
development  effort,  which  inflation  does  not  affect. 

Using  RDT&E  budget  data  from  the  Air  Force’s  E-3  Airborne  Warning  and 
Control  System  (AWACS)  Radar  System  Improvement  Program  (RSIP),  we  show  how 
to  convert  from  the  budget  profile  to  outlays.  Given  the  budget  data,  shown  in  the  top 
row  of  Table  2-1,  we  can  create  an  outlay  profile  in  constant  year  dollars.  We  show  then 
year  dollars  as  TY$M  and  constant  year  dollars  as  CYSOOM. 


Table  2-1.  Example  Budget  Conversion  to  Outlays 


Fiscal  Year 

PhD 

[jggyi 

i W 

~mr 

1W 

~mr 

iwr 

~mr 

ggMfl: 

Budget  (TY$M) 

44.2 

63.7 

68.8 

116.3 

72.1 

30.9 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Outlays  (TY$M) 

22.8 

90.9 

86.8 

53.2 

5.3 

1.8 

0.5 

0.1 

0.0 

Expenditures 

(CYSOOM) 

29.0 

59.9 

72.6 

103.6 

96.3 

57.8 

21.7 

5.6 

1.8 

0.5 

0.1 

0.0 

Since  RDT&E  appropriations  outlay  over  a  period  of  six  years  we  use  (1)  to 
create  an  outlay  profile  in  CYSOOM.  Table  2-2  shows  the  Air  Force  outlay  profile  for 
Research  and  Development  funds.  The  outlay  process  moves  a  portion  of  each  year’s 
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Table  2-2.  Air  Force  RDT&E  Outlay  Rates 


Yrl 

Yr2 

Yr3 

im 

Yr5 

AF  Outlay  Rates 

U222J 

|ggf| 

ms 

Ii2££| 

0.001 

budget  dollars  into  later  years  to  be  expended.  Thus,  budget  dollars  spend  in  later  fiscal 
years  must  reflect  additional  inflation.  To  determine  the  appropriate  budget,  we  convert 
the  constant  year  profile  using  (2)  to  a  then  year  profile  using  the  raw  inflation  index 
shown  in  Table  2-3. 

Table  2-3.  Air  Force  CYOO  RDT&E  Raw  Inflation  Index 


1989 

1990 

1991 

1992 

1993 

1994 

1995 

1996 

1997 

1998 

1999 

2000 

AF  Inflation 

0.787 

0.818 

0.854 

0.877 

0.901 

0.919 

0.937 

0.955 

0.975 

0.982 

0.990 

1.000 

The  Beta  Model 

The  Beta  distribution,  with  its  innate  flexibility,  can  empirically  fit  manpower 
patterns,  the  analog  for  fitting  development  expenditures.  The  probability  density 
function  for  the  Beta  is 


dW(t) 

dt 


^(a  +  b)_ta  1  # (1  _ 0<t<} 

r  (tf)T(h) 


where 


dW(t) 

dt 


represents  that  rate  at  which  development  problems  are  solved,  T(»)  is  the 


gamma  function,  a  >  0,  b  >  0,  and  t  is  time  (Jarvis,  1999).  However,  there  is  little 
justification  for  its  parameter  inputs  (Jarvis,  1999:8).  In  other  words,  the  parameters  do 
not  relate  to  tangible  constructs.  With  no  method  of  estimating  the  model  parameters  a 
and  b  prior  to  program  completion  exists,  the  beta  model  can  fit  the  program  expenditure 
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pattern  only  after  completion  of  the  program.  Therefore,  the  beta  model  provides  limited 
predictive  ability. 


where  a  and  c  are  parameters  selected  to  fit  expenditures  and  t  is  time. 

Similar  to  the  normal  distribution,  the  traits  of  infinite  tails  and  symmetry  about  a 
maximum  characterize  the  Sech-Squared  model  (Parr,  1980:50).  One  of  the  primary 
problems  of  the  Sech-Squared  model  derives  from  its  infinite  tail.  Program  initiation  is 
particularly  problematic  to  define.  Parr  defends  the  notion  of  an  infinite  tail  in  “negative” 
time  (before  zero  on  the  time  scale),  by  stating  that  practical  considerations  will  establish 
the  program  start  time  (Parr,  1980:50).  In  practice,  the  lack  of  a  defined  program  start 
could  have  significant  impact  on  the  predictive  ability  of  the  model.  Compared  to  the 
Rayleigh,  which  starts  at  time  zero  and  is  skewed  right  with  an  infinite  tail,  the  Sech- 
Squared  model  forces  an  amount  of  area  under  the  curve  before  time  zero. 

Every  model  is  an  abstraction  of  reality.  Significant  considerations  in  building  a 
model  are  its  ease  of  use  and  its  ability  to  supply  useful  results.  While  the  Beta  model 
and  the  Sech-Squared  model  can  be  used  to  predict  expenditures,  each  is  flawed  in  a  way 
that  makes  its  usefulness  limited.  While  each  model  has  benefits,  the  Rayleigh  model 
demonstrates  the  greater  functionality. 
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The  Rayleigh  Model 


Norden  related  the  pattern  of  manpower  buildup  and  phase-out  on  development 
efforts  to  a  family  of  mathematical  functions,  called  the  Rayleigh  distribution.  Norden 
developed  the  Rayleigh  model  based  on  skill  level  linearly  increasing  with  time  (Norden, 
1970:122).  Since  the  skill  level  is  linear  with  respect  to  time,  the  rate  of  learning  is 
constant.  The  rate  of  progress  is  proportional  to  the  available  skill  level  and  the  problems 
remaining  to  be  solved  (Jarvis,  1999:9). 

Linear  skills  acquisition  causes  the  initial  ramp-up  in  the  model.  The  Rayleigh 
model  peak  rate  of  expenditures  is  located  at  the  39th  percentile  in  time.  Since  the  rate  of 
work  performed  is  proportional  to  the  work  remaining,  an  exhaustion  of  the  work  causes 
the  exponential  tail.  (Jarvis,  1999:13). 

Putnam  related  the  Rayleigh  model  to  software  development  (Putnam,  1978:33). 

In  testing  the  Rayleigh  model  against  budgetary  data  for  50  Computer  System  Command 
systems,  Putnam  found  that  the  Rayleigh  approximated  the  software  development  well. 

In  fact,  Putnam  checked  the  Rayleigh  model  against  an  additional  150  systems  with  good 
results.  Putnam  asserts  that  the  Rayleigh  model  applies  to  software  development  because 
many  software  programs  follow  the  characteristic  growth  to  a  peak,  then  exponential  fall- 
off  of  the  Rayleigh  (Putnam,  1978:347). 

Other  researches  have  continued  to  apply  the  Rayleigh  model  to  a  wider  variety  of 
situations,  most  notably  DoD  contracts.  Watkins  (1982)  applied  the  Rayleigh  model  to 
Cost/Schedule  Status  Report  data  on  30  DoD  contracts.  Watkins  found  statistically 
significant  relationships  for  Rayleigh  model  between  earned  values  inputs  and  total 
contract  cost  (Watkins,  1982:79).  Later,  Abernathy  (1984)  applied  the  Rayleigh  model  to 
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21  historical  Navy  contracts.  He  was  able  to  adequately  estimate  the  Rayleigh 
parameters  to  fit  the  contract  data  (Abernathy,  1984:37).  While  Abernathy  did  not  meet 
his  intended  goal  of  using  the  Rayleigh  model  as  a  predictive  tool,  he  established  the 
model  as  having  value  for  modeling  DoD  contract  data.  Lee,  Hogue,  and  Hoffman  (Lee, 
1993)  continued  the  application  of  the  Rayleigh  model  to  defense  acquisition  contracts. 
Their  research  showed  that  the  Rayleigh  model  fit  actual  outlays  of  20  defense 
development  contracts. 

An  important  aspect  of  the  research  completed  on  defense  contract  data  is  that  it 
demonstrated  that  a  scaled  Rayleigh  cumulative  density  function  models  development 
expenditures  (Lee,  1997:16).  Based  on  this  assertion,  Lee,  Hogue,  and  Gallagher  spread 
an  R&D  point  estimate  to  a  budget  profile.  Their  contribution  describes  a  method  for 
developing  a  realistic  budget  profile,  based  on  the  Rayleigh  model  of  expenditures  (Lee, 
1997).  Gallagher  and  Lee  (1996)  subsequently  develop  and  test  a  method  of  determining 
final  program  cost  and  completion  time  based  on  Rayleigh  expenditures. 

In  summary,  the  mathematical  relationship  described  in  Norden’s  initial  research 
provides  a  basis  for  many  following  efforts.  Putnam  applied  the  Rayleigh  model  to 
software  development.  Watkins  and  Abernathy  substantiate  the  Rayleigh  model’s 
usefulness  in  modeling  defense  program  expenditures.  Finally,  Lee,  Hogue  and 
Gallagher  (1997)  and  Gallagher  and  Lee  (1996)  provide  useful  tools  for  defense 
development  budget  and  schedule  estimating. 

Rayleigh  Distribution.  The  Rayleigh  model  is  a  subset  of  the  Weibull 

-af 

distribution,  W(t)  =  l- e  U/  ,  with  shape  parameter  J3  =2.  The  concept  that 
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development  projects  involve  solving  a  fixed  number  of  problems,  with  linear  learning, 
provides  the  foundation  for  the  Rayleigh  model.  The  rate  at  which  problems  are  solved  is 
proportional  to  skill  level  and  proportion  of  unsolved  problems  remaining. 


dW(t) 

dt 


pm -wit )] 


where  p(t)  is  the  skills  level  and  IV(t)  is  the  number  of  unsolved  problems.  Integration 
provides  the  total  work  completed  in  the  following  form 


-  \p(r)d(r) 

w(t)  =1-  e  r=0 


Let p(t)  =  ct,  where  c  represents  linear  learning 

w(t)=  l-e-0'5"21 


\-e-05cr 

1  -e 


with  a  =1/2 c,  which  gives  the  Rayleigh  cumulative  distribution  function.  The  Rayleigh 
Rayleigh  cumulative  distribution  function  is  generally  presented  in  the  functional  form 


F(t)  =  l-e“"2  (3) 

Figure  2-1  shows  the  Rayleigh  cumulative  distribution  function  with  a  =  0.94.  When 
scaled  to  a  program’s  total  cost,  the  cumulative  distribution  function  illustrates 
cumulative  expenditures  over  time. 
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Figure  2-1.  Rayleigh  Cumulative  Distribution  Function 

Rayleigh  Probability  Density  Function.  The  derivative  of  the  cumulative 
distribution  (3)  gives  the  probability  density  function.  The  Rayleigh  probability  density 
function  is 

f(t)  =  2atea,!  (4) 

The  scaled  Rayleigh  probability  density  function  provides  the  rate  of  program 
expenditure  for  a  given  time.  Figure  2-2  illustrates  (4)  for  a  value  of  a  =  0.94. 
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1.20 
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Figure  2-2.  Rayleigh  Probability  Distribution  Function 

Infinite  Tail.  While  the  Rayleigh  distribution’s  infinite  tail  is  usually  viewed  as  a 
flaw  of  the  model,  closer  examination  of  the  issue  dismisses  the  problem.  The  normal 
distribution,  the  most  commonly  used  distribution,  shares  the  same  characteristic  of 
infinite  tails.  However,  statisticians  do  not  view  this  as  a  serious  problem.  For  example, 
weight  is  often  modeled  with  a  normal  distribution.  Although  this  implies  that  a 
probability  of  negative  weight  and  infinite  weight  exists,  this  is  overlooked  because  of  the 
incredible  usefulness  of  the  model.  Furthermore,  the  infinite  tail  problem  can  be  avoided 
through  definition.  While  the  tail  stretches  out  to  infinity,  implying  infinite  program 
development  time  in  the  context  of  our  intended  use,  Lee,  Hogue,  and  Gallagher  define 
program  completion  at  97%  of  the  total  program  (Lee,  1 997:6). 

Rayleigh  Parameters.  Both  the  cumulative  distribution  function  shown  in 
Figure  2-1  and  the  probability  density  function  in  Figure  2-2  use  the  shape  parameter  a 
to  determine  the  curve.  The  a  parameter  determines  the  steepness  of  the  cumulative 
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distribution  function.  Figure  2-3  depicts  the  effect  of  a  change  in  the  shape  parameter 
a  on  the  probability  density  function.  Higher  values  of  a  create  a  more  compressed 
curve,  with  an  earlier  peak  time.  This  situation  is  analogous  to  programs  with 
increased  funding  earlier  in  the  program,  producing  earlier  program  completion. 


1.20 


Figure  2-3.  Effect  of  Rayleigh  shape  a  change 

The  Rayleigh  distribution  is  scaled  from  its  value  of  [0,1]  by  multiplying  by  total 
program  expenditures,  d.  Thus  expenditures  are  defined  as  a  function  of  time: 

E(t)=dF(t)=d(\-e-‘"2 )  (5) 

where  d  is  the  total  cost  of  the  program  and  the  t  is  the  time  from  inception  of  the  R&D 
effort  to  completion  (Lee,  1997:31). 

Parameter  Determination.  Different  methods  can  be  used  to  derive  the  Rayleigh 
function  parameters.  The  data  available  generally  dictates  what  method  must  be  used. 
The  two  following  methods  are  characterized  by  the  availability  of  program  data  to 
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determine  the  key  time  parameter  a  and  cost  parameter  d.  The  remaining  function 
parameters  can  be  derived  based  on  either  the  time  that  the  expenditure  rate  is  a 
maximum  or  the  time  that  the  expenditures  are  complete  is  known. 

For  an  ongoing  program,  Lee,  Hogue,  and  Gallagher  indicate  that  if  the  time  of 
the  peak  rate  of  expenditures,  if  known,  provides  a  basis  for  estimating  the  Rayleigh 
parameter  (Lee,  1997:6).  Program  peak  time  is  reasonable  easy  to  specify  for  certain 
types  of  programs.  For  example,  peak  time  is  normally  associated  with  first  flight  on 
aircraft  development  programs  (Lee,  1997:6)  and  full  operational  capability  of  software 
programs  (Putnam,  1978:35). 

From  the  Rayleigh  probability  density  function  in  (4),  the  time  of  peak  time  may 
be  determined  by  finding  the  maximum  of  the  function.  Taking  the  derivative  of  (4) 
produces 

f'(t)  =  2a  e  ~‘"2  +  2at(-2at)e~‘"2 

Setting  equal  to  0  to  find  the  maximum  and  designating  t  as  tp  gives 

0  =  (2a-4a2tp2)e-‘"2 

Since  e~‘"  is  always  positive,  the  coefficient  of  the  exponential  term  must  equal  zero. 
The  result  is 

tp  =  -}=  or  equivalently  a  =  — 

V2a  2  tp 

We  can  derive  the  shape  parameter  a  from  the  peak  time,  tp,  or  vise  versa  since  this  is  an 
one-to-one  function. 

In  the  absence  of  information  about  the  programs  peak  time,  an  alternate  method 
for  deriving  a  is  available.  Often  program  mangers  do  not  know  when  peak  rate  of 
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expenditures  will  occur.  The  final  time  method  is  an  option  to  predict  peak  time,  when 
program  final  time  is  known.  Lee,  Hogue,  and  Gallagher  define  the  final  time,  //,  of  a 
development  project  as  the  time  when  97%  of  the  cumulative  Research  and  Development 
(R&D)  constant  year  expenditures  has  been  reached  (Lee,  1997:32;  Gallagher,  1996:52). 
For  this  analysis,  final  time  // is  defined  as  0.97 d,  or  the  point  at  which  the  program  is 
97%  complete.  Setting  the  total  expenditures  from  (5)  equal  to  the  defined  final  time 
gives 

d(l-e~‘"]  )=  0.97 d 

Solving  for  //gives 

_  lln(0.03) 

,/_V~ ~ 

Since  a  =  Vitp2  ,  we  can  relate  final  time  tf  to  peak  time  tp 

=  fln(0.03) 
f  \  -  0.5/;2 

tf  =  tpJ  ln(0.03)(-2)  =  2.65 tp 

This  relationship  also  equates  to  the  constant  that  the  peak  expenditure  rate  occurs  at  37% 
of  total  program  time. 

While  the  Rayleigh  distribution  has  demonstrated  many  desirable  characteristics, 
including  its  ability  to  model  development  expenditures,  it  proves  inflexible  for  a  broad 
assortment  of  development  programs.  The  Rayleigh  function  forces  a  proportionate  tail 
using  the  peak  expenditure  point  as  the  start.  However,  programs  where  a  proportionate 
tail  is  not  derived  from  the  point  of  peak  expenditures  exist.  For  example,  a  program  may 
have  a  peak  expenditure  followed  by  a  very  short  tail.  The  Rayleigh  distribution  is  not 
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useful  for  providing  an  accurate  model,  in  this  case.  To  better  accommodate  programs  of 
a  variety  of  shapes,  the  Weibull  distribution  can  act  as  a  surrogate  for  the  Rayleigh 
distribution. 

The  Weibull  Distribution 

The  Weibull  distribution  provides  greater  flexibility  in  modeling  program 
expenditures  than  the  Rayleigh  model.  While  the  Rayleigh  model  assumes  program 
funding  and  manpower  start  with  the  inception  of  the  program,  many  programs  do  not 
met  this  assumption.  Some  programs  begin  with  one  or  more  years  of  insignificant 
funding,  while  some  programs  may  have  large  initial  funding  that  tapers  off.  The  three- 
parameter  Weibull  distribution  handles  either  scenario.  The  Weibull  accommodates 
insignificant  initial  funding  with  a  location  parameter.  The  Weibull,  with  a  flexible  shape 
parameter,  accommodates  other  expenditure  profiles  not  approximated  well  by  the 
Rayleigh,  such  as  most  funding  delays. 

We  derive  the  Weibull  cost  model  from  the  assumption  that  the  rate  at  which 
work  is  completed  is  a  function  of  performance  and  remaining  work.  Define  the  percent 
work  remaining  as  w{t)  and  performance  as  pit),  both  at  time  t.  Then 

^  =  />(')[!  -*</)], 

at 

which  may  be  solved  for  w(t).  Let  z(t )  =  1  -  w(t) ,  so  —  and 

Cl^l  =  -p{t)z(t). 
at 
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Integrating,  we  obtain 


ln(z(/))  =  -  \p<S)dr. 

r=0 

We  evaluate  both  sides  to  the  power  of  the  base  of  the  natural  logarithm, 


-  \p(r)dT 

z(t )  =  e  r=0 


We  substitute  back  in  percent  of  work  to  obtain 

-  iP(r)‘lr 

w{t)=\-e 

We  define  performance  on  the  program  for  any  given  time  as  a  constant  multiplied  by 

T=f  T=1  q 

time  to  constant  power,  p(t)  =  cth .  Since  \pir)dr=  \crhdz= - tM , 

r=0  r=0  b  + 1 


w{t)  =  \-e 


_E_th+ 1 
Zh-1 


With  linear  growth  in  performance  over  time,  b=  1  and  a  =  c/2,  we  obtain  the  Rayleigh 
cumulative  distribution  function  shown  in  (1).  If  performance  improves  with  time  to  a 
power  other  than  one,  we  have  derived  percent  work  complete  according  to  a  Weibull 
cumulative  distribution  function. 

The  Weibull  cumulative  distribution  function  is  generally  written  as 

-f— T 

F(t)  =  \-e{s)  (6) 

while  the  Weibull  probability  density  function  is  written  as 


m= 


P(‘-rv 
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where  t  =  time,  y=  location,  5  =  scale,  and  /?  =  shape  (Hines,  1980:165).  The  shape 
parameter  (3  =  b  + 1  >0  and  the  scale  parameter  S  =  (a+i/«)^+i>  0.  For  (3=2,  the  Weibull 
is  the  Rayleigh  distribution. 

When  the  Weibull  location  parameter  y  =  0  and  the  Weibull  shape  parameter  /?= 

2  ,  (3)  and  (6)  are  equivalent.  The  Rayleigh  scale  parameter  in  (3),  a ,  is  equal  to  1/52  in 
(6).  The  Weibull  shape  parameter  allows  skills  acquisition  at  other  than  linear  rates. 

Also,  the  Weibull  time  of  peak  expenditures  does  not  fix  the  completion  time  of  the 
program. 

We  represent  the  Weibull  cost  model  as  the  cumulative  distribution  function  in  (6) 
multiplied  by  a  cost  scalar  d  with 


E(t)  =  d 


1  -e 


(7) 


For  given  shape,  scale,  and  location  parameters,  the  Weibull  cost  model  gives  the 
cumulative  cost  of  a  program  at  specified  time  t. 

Weibull  Parameters.  The  flexibility  of  the  Weibull  distribution  allows  it  to 
approximate  a  variety  of  different  distributions.  Changing  the  Weibull  parameters 
produce  noteworthy  changes  in  the  distribution.  Figure  2-4  details  the  effect  of  changing 
the  Weibull  scale  parameter  Awhile  holding  the  other  parameters  constant.  Increasing 
5  from  1 .0  to  1.3,  elongates  the  program.  A  scale  increase  implies  that  the  program  will 
take  longer  to  accomplish. 
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Figure  2-4.  Effect  of  Weibull  scale  £  change,  with  J3=  2,  y =0 

The  Weibull  shape  parameter  is  interesting  because  it  has  the  greatest  affect  on 
the  overall  distribution  appearance.  Different  shape  parameter  values  allow  the  Weibull 
distribution  to  approximate  other  distributions.  As  noted  earlier,  a  Weibull  with  a  shape 
parameter  / 3  =  2.0  is  a  Rayleigh  distribution.  The  Weibull  with  /?=  1  is  an  exponential 
distribution.  For  /?=  3.4,  the  Weibull  approximates  a  normal  distribution. 

In  programmatic  terms,  the  shape  parameter  determines  the  location  of  peak 
spending  for  the  program,  for  given  location  and  scale  parameters.  This  aspect  of  the 
model  is  important,  since  peak  spending  corresponds  to  a  significant  development  event, 
such  as  first  flight.  Figure  2-5  shows  the  effect  of  changing  the  Weibull  shape  parameter 
P,  holding  the  other  parameters  constant. 
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Figure  2-5.  Effect  of  Weibull  shape  (3  change,  with  8=  1,  y-  0 


The  location  parameter  y  allows  the  Weibull  distribution  to  shift  along  the  x-axis. 
In  certain  program  situations,  the  first  several  budget  years  contain  very  little  funding. 

For  program  data,  the  location  parameter  identifies  the  length  of  time  before  the  program 
significantly  started.  Figure  2-6  depicts  the  effect  of  changing  the  location  parameter, 
with  all  other  parameters  held  constant. 
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Figure  2-6.  Effect  of  Weibull  location  ^change,  with  >9=1,  8-  2 
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Factors  Contributing  to  Development  Cost 


Figure  2-7.  Factors  that  Contribute  to  Development  Cost 


A  study  by  Belcher  and  Dukovich  (2000)  provides  a  macro  perspective  on  the 
contributors  to  the  cost  of  development  programs.  Based  on  expert  opinion,  Belcher  and 
Dukovich  determine  a  framework  of  how  development  programs  incur  cost.  They  cite 
three  major  areas  that  cause  development  cost:  productivity,  scope  and  economic.  They 
refine  the  detail  by  providing  several  factors  that  span  each  major  area.  In  total  Belcher 
and  Dukovich  identify  1 2  factors  that  contribute  to  development  cost. 

In  this  study,  we  examine  one  of  the  factors  described  by  Belcher  and  Dukovich. 
Our  models  focus  only  on  the  impact  of  the  “Funding  Constraints”  factor  on  cost  and 
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schedule  growth.  We  measure  the  funding  constraints  by  how  well  the  initial  budget 
profile  supports  a  Rayleigh  distributed  expenditures. 

Chapter  Summary 

Based  on  research  conducted  over  the  past  30  years,  the  Rayleigh  model  provides 
a  method  for  determining  an  expenditure  profile  for  development  programs.  Norden 
described  a  useful  mathematical  relationship  for  development  effort  and  manpower  over 
time.  Putnam  applied  the  Rayleigh  model  to  software  development.  Lee,  Hogue,  and 
Gallagher  concluded  the  Rayleigh  modeled  outlays  of  a  wide  variety  of  defense 
development  programs.  Therefore,  using  the  Rayleigh  and  its  generalization,  the  Weibull 
function,  is  justified  in  modeling  expenditure  profiles.  After  describing  the  Rayleigh,  its 
derivation,  and  its  attributes,  this  chapter  discusses  the  flexible  Weibull  distribution.  We 
derive  the  applicability  of  the  Weibull  model  to  development  programs,  explain  its 
parameters,  and  describe  its  ability  to  model  a  wide  variety  of  funding  profiles.  Finally, 
we  provide  a  summary  view  of  the  twelve  factors  that  affect  program  growth  while 
noting  that  we  are  only  assessing  the  impact  of  one  factor,  funding  constraints. 
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III.  Research  Methodology  and  Results 


Chapter  Overview 

Previous  research  indicates  that  the  Rayleigh  cumulative  distribution  function 
models  contract  expenditures  effectively.  This  research  intends  to  expand  the  application 
of  the  model  to  include  program  level  expenditures,  a  much  broader  scale.  The  focus  of 
this  chapter  is  to  explain  the  methodology  employed  to  test  for  a  relationship  between 
program  initial  funding  profiles  and  growth. 

The  initial  step  of  this  research  is  to  collect  appropriate  data.  We  collected 
program  budget  data  from  the  Selected  Acquisition  Report,  maintained  by  the  Office  of 
the  Secretary  of  Defense  Program  Analysis  and  Evaluation  (PA&E).  We  selected 
programs  meeting  certain  criteria  from  this  large  data  repository. 

After  translating  the  initial  budget  data  into  constant  year  expenditures,  we 
estimated  parameters  for  both  a  Weibull  and  Rayleigh  cost  models.  We  then  evaluate 
whether  or  not  the  expenditures  from  the  initial  budget  fits  a  Rayleigh  distribution  via  the 
Cramer  Von  Mises,  Anderson-Darling,  and  Kolmolgorov-Smimov  goodness  of  fit 
statistics.  Each  of  these  parameters  and  statistics  is  available  as  an  independent  variable 
for  regression  modeling. 

We  test  for  a  relationship  between  program  growth  by  creating  a  regression 
model.  We  create  two  separate  models,  one  for  cost  growth  and  one  for  schedule  growth, 
using  multiple  linear  regression.  We  validate  the  resulting  models  by  assessing  the 
respective  predictive  abilities  against  programs  not  included  in  the  initial  dataset. 
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Program  Data 


We  collected  program  funding  data  for  evaluating  the  research  hypothesis  from 
numerous  Selected  Acquisition  Reports  (SARs).  The  full  dataset  consists  of  thirty-seven 
Army,  Navy,  Air  Force,  and  Joint  programs.  The  program  total  costs  range  in  constant 
year  dollars  (CY00$)  from  $15.5M  to  $13,686.3M. 

Selected  Acquisition  Report  (SAR).  The  SAR  provides  standard,  comprehensive 
summary  reporting  of  cost,  schedule,  and  performance  information  for  DoD  Acquisition 
Category  (ACAT)  IC  and  ID  Major  Defense  Acquisition  Programs  (MDAPs)  to 
Congress.  The  SAR  compares  the  current  estimate  of  total  program  acquisition  cost, 
schedule,  and  performance  data  against  a  baseline  to  derive  program  cost  variances  and 
analysis.  The  SAR  also  includes  unit  cost  reporting  and  Nunn-McCurdy  unit  cost  breach 
information  in  accordance  with  Title  10  USC  2433  for  programs  beyond  Milestone  II 
(DAD,  1999:2.B.3.2).  The  SAR  reports  are  prepared  annually  in  conjunction  with  the 
President's  budget.  Quarterly  exception  reports  are  required  only  for  those  programs 
experiencing  unit  cost  increases  of  at  least  15  percent  or  schedule  delays  of  at  least  six 
months.  Programs  submit  Quarterly  SARs  for  initial  reports,  final  reports,  and  for 
programs  rebaselined  at  major  milestone  decisions. 

The  total  program  cost  estimates  provided  in  the  SARs  include  research  and 
development,  procurement,  military  construction,  and  acquisition-related  operation  and 
maintenance.  Total  program  costs  reflect  actual  costs  to  date  as  well  as  anticipated  costs 
for  future  efforts.  All  estimates  include  allowances  for  anticipated  inflation;  the  SAR 
presents  them  in  millions  of  then  year  dollars  (TY$M). 
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Over  the  past  3  decades,  the  SAR  format  changed  several  times.  In  1983,  a  SAR 
Improvement  Task  Force  significantly  reduced  the  content  of  the  SAR.  However,  in 
1985  the  FY86  Authorization  Act  restored  the  information  that  had  been  removed  and 
added  production  rate  and  operating  and  support  cost  information.  The  FY87 
Authorization  Act  provided  for  limited  reporting  for  pre-Milestone  II  programs  and 
relaxed  some  reporting  criteria.  The  FY92  Authorization  Act  gave  the  Secretary  of 
Defense  the  authority  to  waive  selected  acquisition  reporting  for  certain  programs  and  to 
change  the  content  of  the  SAR  as  long  as  the  appropriate  House  and  Senate  committees 
were  notified  in  advance.  The  Federal  Acquisition  Streamlining  Act  of  1994  changed  the 
baseline  for  unit  cost  reporting  purposes  from  the  prior  President’s  budget  to  the 
approved  acquisition  program  baseline  and  substituted  procurement  unit  cost  for  current 
year  procurement  unit  cost.  In  1 996,  the  Department  made  additional  changes  in  SAR 
format  and  content,  which  reduced  the  volume  of  the  SAR  by  about  20-30%. 

Program  Selection.  The  Office  of  the  Secretary  of  Defense’s  (OSD)  SAR  archive 
provides  funding  data  on  hundreds  of  defense  programs,  dating  back  to  the  early  1970s. 
However,  not  all  of  the  programs  are  useful  to  this  study.  The  research  requirements  and 
the  SAR  database  attributes  limit  the  number  of  applicable  programs. 

To  establish  a  proper  funding  profile,  the  identified  program’s  budget  needs  to  be 
represented  in  annual  budget  format.  Before  1 980,  the  SAR  reported  all  budget 
information  in  aggregate.  While  interesting,  the  cumulative  budget  and  expenditures  do 
not  provide  adequate  detail  to  construct  a  funding  profile.  Around  1982,  OSD  changed 
the  SAR  format  to  include  annual  budget  and  expenditure  data.  We  selected  only 
programs  that  reported  in  the  annual  budget  format. 
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Since  the  goal  of  this  study  is  to  establish  a  relationship  between  initial  budget 
and  program  growth,  the  dataset  necessarily  must  include  only  completed  programs.  This 
requirement  forces  the  final  program  S AR  to  be  dated  1 998  or  earlier,  since  we  used  the 
1999  version  of  the  OSD  SAR  database.  The  combination  of  the  first  two  criteria  bound 
programs’  development  life  between  1982  and  1998.  We  excluded  all  of  the  programs 
falling  out  of  this  range. 

Joint  programs  are  generally  large  programs  with  funding  from  various  sources, 
including  the  services  and  other  defense  agencies.  We  prefer  to  evaluate  each  funding 
source  of  the  program  separately,  to  fit  the  funding  profiles  to  the  distributions  at  a  lower 
level.  However,  the  apparent  financial  management  of  joint  programs,  combined  with 
the  top-level  SAR  data  collection  precludes  separate  evaluation.  Therefore,  we  treat  joint 
programs  as  a  single  program  by  summing  individual  service  contributions,  avoiding  the 
problems  of  discerning  interservice  funding  transitions. 

For  each  program,  we  collect  the  initial  and  final  budget  profile.  Each  budget 
profile  includes  the  then  year  funding  by  year.  The  SAR  presents  funding  in  both  then 
year  and  base  year  formats.  However,  we  collect  the  then  year  information  to  apply 
conversion  to  constant  year  consistently,  across  programs.  Table  3-1  shows  an  example 
budget  profile  for  the  Air  Force’s  E-3  Airborne  Warning  and  Control  System  (AWACS) 
Radar  System  Improvement  Program  (RSIP). 


Table  3-1.  RSIP  Program  Budget  Profile 
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Table  3-2  shows  the  programs  included  in  the  study  and  their  associated  dollar 
cost  overrun,  schedule  slip,  initial  budget  dollars,  and  initial  budget  profile  years.  We 
show  cost  growth  as  the  dollar  difference  between  the  initial  program  cost  and  final 
program  cost,  expressed  in  CY$00M.  We  show  schedule  growth  as  a  ratio.  For 
example,  if  the  original  program  was  5  years  long  and  the  final  program  was  7  years  long, 
the  schedule  slip  is  1.40.  A  schedule  slip  of  1 .00  indicates  that  the  final  program  was  the 
same  length  as  the  initial  program,  indicating  no  growth.  The  37  programs  in  Table  3-2 
show  a  range  of  schedule  growth  of  0.67  to  3.20,  with  an  average  schedule  growth  of 
1.26.  The  programs  range  in  schedule  growth  from  -$688.62M  to  $1 199.33M,  with  an 
average  cost  growth  of  $40.33M  CY00. 


3-5 


Table  3-2.  Included  Programs 


Program 

Cost  Overrun 
(CY00SM) 

schedule 

Slip 

Ratio 

L,ost  initial 
Budget 
(CY00$M) 

Years  Initial 
Budget 

1.31 

mmiiKi 

. . n 

2  ASPJ-J 

42.912 

1.13 

752.7 

15 

3  B-1B 

1199.329 

1.43 

3861.3 

7 

4  Battleship 

2.619 

mmm 

32.2 

6 

5  IUS 

21.987 

m 

1230.7 

15 

6  KC-135R 

-3.694 

nro 

8 

7  Kiowa 

-0.588 

1.17 

339.9 

6 

8  Lantirn 

152.199 

1.50 

654.3 

10 

9  Trident-MSL 

-688.615 

1.23 

13686.3 

13 

10  Trident-Sub 

22.154 

3.20 

78.9 

5 

11  RPV(Aquila) 

39.137 

TOO- 

11219 

14 

12  MK48  ADCAP 

101.786 

1.23 

1522.9 

13 

13  E-6(Tacamo) 

32.614 

1.13 

497.5 

8 

14  Avenger 

1.398 

1.00 

15.5 

3 

15  PLS 

0.306 

1.80 

51.9 

5 

16  RSIP 

27.620 

1.33 

449^ 

6 

17  Longbow 

180.750 

1.22 

847.9 

9 

18  CMU 

174.181 

1.10 

1511.1 

20 

19  AOE6 

2.451 

1.11 

41.1 

9 

20  TRITAC 

13.737 

1.22 

576.7 

18 

■  11  II  Mil  1  M 

-28.432 

— — 

22  JSOW 

31.381 

1.44 

404.2 

23  ASAT 

276.789 

0.89 

2112.4 

24  ADDS 

87.449 

1.78 

267.5 

25  LCAC 

12.892 

1.71 

37.8 

7 

26  LSD41 

-5.515 

9T4- 

11 

27  MK  50 

77.069 

1.18 

2129.8 

17 

28  Backscatter 

-48.563 

1.09 

642.2 

11 

29  Peacekeeper 

-576.644 

1.57 

9847.5 

7 

30  T46A 

-57.526 

0.80 

546.0 

10 

31  Cvheio 

ruo- 

snr 

4 

32  TA01870iler 

-1.890 

0.67 

24.7 

9 

33  FDS 

479.552 

1.30 

931.6 

10 

34  ATARS 

-7.870 

1.00 

203.0 

9 

35  SRAMII 

-347.508 

0.89 

1337.4 

9 

36  ATACMS 

-300.523 

1.00 

1058.3 

11 

37  TOW2 

61.400 

1.44 

155.8 

9 
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Conversion  to  Outlays 


Theory  and  past  research  suggest  that  outlays  should  follow  a  Rayleigh 
distribution.  We  convert  the  program  budget  profile  into  an  expenditure  profile  to  which 
we  fit  Rayleigh  and  Weibull  models.  The  conversion  process  takes  two  steps:  converting 
budget  dollars  to  outlays  and  adjusting  for  inflation.  We  depict  the  RSIP  program  budget 
in  Figure  3-1. 


Figure  3-1.  Then  Year  Budget  Profile  for  RSIP  Program 
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The  first  step  in  the  translation  to  an  appropriate  outlay  profile  is  to  convert  the 
then  year  budget  profile  into  outlays.  We  apply  the  outlay  rates  the  historical  percents  of 
funds  obligated  in  the  budget  year  and  subsequent  years.  After  multiplying  the  budget  by 
the  outlay  rates,  we  summed  the  funds  obligated  from  various  budget  years  to  obtain  the 
amount  obligated  in  each  fiscal  year,  as  shown  in  (1). 

The  OSD  Comptroller,  OSD(C),  provides  Congressionally  approved  outlay  rates, 
also  called  spendout  rates,  annually  for  service  and  appropriation.  OSD(C)  develops 
outlay  rates  for  every  funding  appropriation.  These  outlay  rates  reflect  the  percent  of 
budget  authority  expected  to  be  expended  by  year,  based  on  historical  experience.  Outlay 
rates  are  implicitly  considered  in  the  budgetary  concept  of  then-year  dollars.  OSD(C) 
disseminates  outlay  rates  within  DoD  for  development  of  weighted  inflation  indices. 
Historically,  different  commodities  have  had  different  rates  of  inflation.  Within  the  DoD, 
the  rates  of  inflation  are  linked  to  the  source  appropriation,  such  as  Military  Construction 
(MILCON)  or  Research  Development  Test  and  Evaluation  (RDT&E).  DoD  weights  the 
inflation  by  the  outlay  rates  to  form  weighted  composite  inflation  indices.  In  addition  to 
different  inflation  rates,  the  appropriations  do  not  all  spent  at  the  same  rate.  Military 
Personnel  (MILPERS)  costs  are  spent  in  the  year  appropriated,  while  Research 
Development  Test  and  Evaluation  costs  may  be  spread  out  over  as  much  as  six  years. 
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Table  3-3.  Service  R&D  Outlay  Rates  (Percentages) 


Air  Force 


FY01 

FY00 

FY99 

FY98 

FY97 

FY96 

FY95 

FY94 

FY93 

Avg 

Yrl 

wr 

~  WT 

59.1 

wr 

TO“ 

WT 

30“ 

WT 

50.8 

51. 56 

Yr2 

33.7 

34.5 

33.1 

37.4 

39.9 

39.1 

38.8 

38.8 

34.5 

36.64 

Yr3 

3.6 

3.6 

5.3 

6.8 

8.9 

8.9 

8.8 

8.9 

9.5 

7.14 

Yr4 

1.0 

1.0 

1.4 

3.0 

3.6 

3.6 

3.6 

3.6 

3.4 

2.69 

Yr5 

0.3 

0.3 

0.4 

0.8 

1.1 

1.1 

1.1 

1.1 

1.1 

0.81 

Yr6 

0.0 

0.3 

0.3 

0.3 

0.3 

0.4 

0.4 

0.4 

0.4 

0.31 

Yr7 

0.0 

0.0 

0.2 

0.0 

0.1 

0.0 

0.2 

0.0 

0.2 

0.08 

98.1 

90“ 

wr 

99.0 

99.7 

99T" 

993“ 

99T" 

99^ 

99.23 

Army 


FY01 

FY00 

FY99 

FY98 

FY97 

FY96 

FY95 

FY94 

FY93 

Avg 

Yrl 

57.5 

50"" 

50“ 

50“ 

50“ 

50“ 

50“ 

50 

56.92 

Yr2 

32.5 

33.7 

33.0 

33.0 

33.0 

34.0 

34.0 

34.0 

34.0 

33.47 

Yr3 

6.3 

5.0 

5.3 

5.3 

5.3 

5.3 

5.3 

7.3 

7.3 

5.82 

Yr4 

2.1 

2.1 

2.1 

1.8 

1.8 

1.8 

1.8 

1.8 

1.8 

1.90 

Yr5 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

0.80 

Yr6 

0.0 

0.5 

0.5 

0.5 

0.5 

0.5 

0.5 

0.5 

0.5 

0.44 

Yr7 

0.0 

0.0 

0.2 

0.0 

0.2 

0.0 

0.2 

0.0 

0.2 

0.09 

99T" 

90“ 

991T 

991“ 

99.6 

993“ 

993“ 

993“ 

"99.6 

99.44 

Navy 

FY01 

FY00 

FY99 

FY98 

FY97 

FY96 

FY95 

FY94 

FY93 

Avg 

Yrl 

wr 

WT~ 

W5~ 

50“ 

55.9 

55.9 

541T 

55ir 

55.0 

SOT 

Yr2 

31.4 

33.6 

32.5 

33.1 

31.5 

31.5 

32.4 

33.4 

33.4 

32.53 

Yr3 

5.9 

4.5 

4.5 

5.4 

8.2 

8.2 

8.0 

7.8 

7.8 

6.70 

Yr4 

1.9 

1.0 

1.0 

2.0 

2.0 

2.0 

2.0 

1.3 

1.3 

1.61 

Yr5 

0.7 

0.3 

0.3 

0.6 

1.1 

1.1 

1.1 

1.1 

1.1 

0.82 

Yr6 

0.0 

0.2 

0.2 

0.2 

0.4 

0.4 

0.4 

0.4 

0.4 

0.29 

Yrl 

0.0 

0.0 

0.0 

0.0 

0.6 

0.0 

0.2 

0.0 

0.2 

0.11 

99.4 

90“ 

WJT 

9973“ 

9977“ 

99T" 

90“ 

993“ 

993 

WM 

Since  the  outlay  rates  vary  slightly  from  year  to  year,  we  created  an  average 
outlay  rate  profile  for  each  of  the  services.  We  apply  the  average  outlay  profile  to  each 
program  for  a  given  service.  The  composite  outlay  rate  profile  is  the  average  of  nine 
outlay  profiles  from  FY93  to  FY01.  Table  3-3  shows  the  yearly  outlay  rates  and  the 
composite  average  used  in  the  individual  program  outlay  calculations. 
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Figure  3-2.  Then  Year  Budget  and  Outlay  Profile  for  RSIP  Program 

For  the  RSIP  budget  example  in  Figure  3-1,  we  apply  the  Air  Force  average 
outlay  rates  and  sum  to  determine  outlays  for  each  fiscal  year.  Figure  3-2  depicts  the 
resulting  outlays,  also  called  expenditures,  from  the  RSIP  initial  budget.  Table  3-4 
provides  the  actual  results  for  the  RSIP  conversion  to  outlays. 


Table  3-4.  RSIP  Budget  to  Outlays 


Fiscal  Year 

12221 

~mr 

1994 

1995 

1996 

1997 

Uggjl 

Budget  (TY$M) 

44.2 

63.7 

68.8 

116.3 

72.1 

30.9 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Outlays  (TY$M) 

22.8 

49.0 

62.0 

90.9 

86.8 

53.2 

20.4 

5.3 

1.8 

0.5 

0.1 

0.0 
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Convert  Then  Year  to  Constant  Year.  After  converting  the  then  year  budget 


profile  to  a  then  year  outlay  profile,  we  translate  into  the  outlays  into  constant  year 
dollars.  This  translation  makes  the  funds  spent  in  various  years  comparable  within  a 
program  and  across  programs  by  eliminating  the  impact  of  inflation.  For  instance,  $10 
million  dollars  in  1990  is  worth  more  than  $10  million  in  2000  dollars,  because  inflation 
has  eroded  the  value  of  a  dollar  during  that  time.  Also,  the  Rayleigh  theory  is  based  on 
linear  skills  acquisition,  which  does  not  have  an  inflation  component.  Thus,  we  must 
remove  the  effects  of  inflation  from  the  outlays  to  properly  fit  the  model. 

Since  all  of  the  program  data  is  in  then  year  dollars,  we  convert  each  year  of 
budget  to  constant  year  using  the  raw  inflation  index  for  the  appropriate  service  and 
appropriation.  Like  outlay  rates,  OSD  publishes  inflation  rates  for  each  service  annually. 
The  services  sometimes  modify  published  rates  to  account  for  service  peculiarities. 
Appendix  A  contains  tables  for  the  Air  Force,  Army  and  Navy  raw  and  Weighted 
Inflation  indices. 

To  apply  the  Rayleigh  model  properly,  we  convert  then  year  outlay  profiles  to 
constant  year  2000  profiles.  Since  each  program’s  outlays  are  in  then  year  dollars,  a 
single  application  of  the  appropriate  service  inflation  rates  converts  the  outlays  to 
constant  year  2000  dollars.  Table  3-5  shows  results  of  the  conversion  to  constant  year 
dollars  for  the  Air  Force  RSIP  program. 


Table  3-5.  RSIP  CY00$M  Expenditures 


Fiscal  Year 

rm 

~mr 

~mr 

T5T 

TWs 

Budget  (TY$M) 

44.2 

63.7 

68.8 

116.3 

72.1 

30.9 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Outlays  (TY$M) 

22.8 

49.0 

62.0 

90.9 

86.8 

53.2 

20.4 

1.8 

0.5 

0.1 

0.0 

Expenditures 

(CY$00M) 

29.0 

59.9 

72.6 

103.6 

96.3 

57.8 

21.7 

1.8 

0.5 

0.1 

0.0 
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Figure  3-3  contrasts  the  initial  then  year  budget  profile  to  the  constant  year  2000 
expenditure  profile  for  the  RSIP  program. 
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u 

W) 

£  60.0 

CL 

1  40.0 

20.0 

0.0 

1989  1990  1991  1992  1993  1994  1995  1996  1997  1998  1999  2000 

Fiscal  Year 

Figure  3-3.  Constant  Year  Expenditure  Profile  for  RSIP 

With  the  expenditure  profile  in  constant  year  2000  dollars,  we  fit  a  Rayleigh  and 
Weibull  distribution  to  the  expenditures  using  the  least  squares  error  technique  to  identify 
the  respective  parameters.  The  purpose  of  this  is  threefold.  The  Rayleigh  parameters 
provide  a  means  to  evaluate  whether  the  profile  is  Rayleigh  using  Goodness  of  Fit 
statistics.  The  Weibull  parameters  provide  another  means  of  evaluating  whether  the 
expenditures  are  Rayleigh,  by  examining  the  shape  parameter.  Finally,  we  use  the 
Rayleigh  parameters,  Weibull  parameters  and  Goodness  of  Fit  statistics  as  a  pool  of 


3-12 


independent  variables  to  create  a  regression  model.  The  regression  model  establishes  the 
relationship  between  program  growth  and  initial  budget  profile. 

Parameter  Estimation 

Fitting  the  Rayleigh  and  Weibull  functions  to  the  expenditures  involves 
identifying  the  parameters  for  each  respective  function  that  minimizes  the  sum  of  squared 
difference  between  the  actual  cumulative  CY00$M  expenditures  from  (1)  and  (2)  and  the 
proposed  theoretical  expenditure  model,  either  Rayleigh  or  Weibull  given  in  (5)  and  (7), 
respectively. 

For  each  year,  we  subtract  the  scaled  Rayleigh  cumulative  distribution  function 
(5)  value  from  the  actual  expenditures.  This  difference  is  squared  and  added  to  each  of 
the  other  squared  differences.  We  use  Solver™  to  minimize  the  sum  of  the  squared 
errors  by  changing  the  Rayleigh  cumulative  distribution  function  parameters.  Let  C„ 
represent  the  cumulative  actual  expenditures  expressed  in  CY00$M  at  the  end  of  the  «th 
fiscal  year  (time  /„).  We  represent  the  least  squares  estimation 

n=N 

min  Z(Error)2  =  min  ^ 

»= i 

where  /?  =  2  for  the  Rayleigh  model  and  is  allowed  to  vary  for  the  Weibull  approximation 
and  N  is  the  number  of  budget  years  plus  outlay  years  minus  one. 

The  Rayleigh  has  difficulty  modeling  programs  with  insignificant  funding. 
Therefore,  we  allow  y  to  vary  for  the  Rayleigh  distribution  to  take  advantage  of  the 
location  parameter’s  ability  to  model  programs  with  very  low  initial  funding.  Including 


Cn~d 


l-e 


(!?n 


(8) 


3-13 


the  location  parameter  allows  for  better  fit,  and  a  lower  rejection  with  the  goodness  of  fit 
tests  for  programs  being  estimated  with  the  Rayleigh. 

Table  3-6  shows  the  iterations  from  budget  to  cumulative  then  year  expenditures 
for  the  Air  Force  RSIP  program.  We  estimate  the  parameters  for  the  Rayleigh  function 
and  Weibull  function  that  minimize  the  sum  of  squared  error. 


Table  3-6.  Cumulative  Expenditure  Profile  for  the  RSIP  Program 


Fiscal  Year 

1989 

1990 

1999 

Budget  (TY$M) 

44.2 

63.7 

68.8 

116.3 

72.1 

30.9 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Outlays  (TY$M) 

22.8 

49.0 

62.0 

90.9 

86.8 

53.2 

20.4 

5.3 

1.8 

0.5 

0.1 

0.0 

Expenditures 

(CYSOOM) 

29.0 

59.9 

72.6 

103.6 

96.3 

57.8 

21.7 

5.6 

1.8 

0.5 

0.1 

0.0 

Cum  Expenditures 
(CYSOOM) 

29.0 

88.9 

161.5 

265.1 

361.4 

419.2 

441.0 

446.5 

448.3 

448.8 

448.9 

448.9 

Table  3-7  depicts  the  minimization  results  for  the  RSIP  program,  when  /?=  2  in  (13).  For 
any  other  combination  of  parameters,  the  sum  of  squared  error  would  be  greater  than 
1136.97. 


Table  3-7.  RSIP  Minimization  of  Z(Error)2 


1989 

~~r wr 

~mr 

1W 

1995 

1996 

1W 

nw 

"  1'999 

jgj 

12.6 

66.0 

98.7 

99.5 

78.0 

49.9 

26.6 

11.9 

4.5 

1.5 

0.4 

0.1 

12.6 

78.6 

177.3 

276.8 

354.8 

404.7 

431.2 

443.1 

447.7 

449.2 

449.6 

449.7 

Error 

267.7 

Igg2*l 

249.2 

136.8 

43.3 

2l  1.8 

94.8 

11.4 

mm 

IBSi 

1136.97 
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Table  3-8  shows  the  initial  parameter  values,  along  with  the  parameters  resulting  from  the 
minimization  in  (8).  The  start  value  for  d  is  1 .05  times  the  initial  total  budget.  We  set  8 
to  7.0,  since  the  parameter  estimation  failed  for  lower  values.  We  set  y  to  zero,  to  reflect 
a  presumed  program  start  at  time  zero.  We  allow  the  Rayleigh  to  use  a  location 
parameter,  similar  to  the  Weibull,  allowing  for  greater  flexibility  in  fitting  programs  with 
insignificant  initial  funding. 


Table  3-8.  RSIP  Rayleigh  Parameters 


Start  Values 

Least  Squares 
Rayleigh  Parameters 

d= 

Cost  Factor 

472.3 

d  =  449.70 

5  = 

Scale  (  >0) 

7.0 

5  _  3.708 

a  =  0.073 

Shape 

B  =  2 

Y  = 

Location  (  >t,  else  0) 

0.0 

Y=  0.375 
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Figure  3-4  depicts  the  resulting  Rayleigh  model  fit  to  both  the  cumulative  and  rate 
functions.  While  the  sum  of  squared  error  technique  produces  the  closest  Rayleigh  model 
for  given  expenditures,  it  does  not  have  the  flexibility  to  fit  expenditures  exactly. 
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Figure  3-4.  RSIP  Cumulative  (top)  and  Rate  (bottom)  Rayleigh  Models 
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We  use  the  same  procedure  to  identify  the  parameters  for  the  Weibull  model. 
Table  3-9  shows  the  Weibull  least  squares  parameters,  with  a  sum  of  squared  error  of 
636. 1 7,  for  the  RSIP  program.  Since  the  flexibility  of  the  Weibull  model  allows  for 
closer  approximation  of  the  expenditures,  the  sum  of  squared  error  technique  produces  a 
smaller  value. 


Table  3-9.  RSIP  Weibull  Parameters 


Start  Values 

Least  Squares 
Weibull  Parameters 

d  - 

Cost  Factor 

472.3 

d=  449.29 

5  = 

Scale  ( >0) 

5.0 

§  _  4.103 

P  = 

Shape  ( >0) 

1.5 

P  =  2.359 

Y  = 

Location  ( >t,  else  0) 

0.0 

y-  0.000 
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Figure  3-5  depicts  the  resulting  Weibull  model  fit  to  both  the  cumulative  and  rate 
functions.  In  conjunction  with  a  smaller  sum  of  squared  error,  the  Weibull  model  has  a 
closer  visual  fit  to  the  expenditures  than  the  Rayleigh  model. 
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Figure  3-5.  RSIP  Cumulative  (top)  Rate  (bottom)  Weibull  Models 
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Goodness  of  Fit 


To  evaluate  if  actual  expenditures  are  Rayleigh,  we  assess  the  cumulative  proportion 
of  the  actual  expenditures  against  the  Rayleigh  cumulative  distribution  function.  This 
comparison  describes  a  statistical  technique  called  “goodness  of  fit.”  We  use  three 
goodness  of  fit  statistics. 

The  Kolmolgorov  is  simply  the  maximum  absolute  deviation  between  F„{x),  which  is 
the  empirical  distribution  function  (EDF),  and  F0(x),  which  is  the  hypothesized 
cumulative  distribution  function  (CDF). 

K  -  max|F„  (x)  -  F0  (x)| 

The  Cramer-von  Mises  and  the  Anderson-Darling  “goodness  of  fit”  statistics  may  be 
represented  in  the  form 


Q  =  n  J[Fn(x)-F0(x)]2T(xyFu(x) 

X=-o0 

where  W(x)  is  a  weighting  function,  and  n  is  the  number  of  data  points.  When  F(x)  =  1 , 
the  Cramer-von  Misses  statistic  W1  =  Q.  The  Anderson-Darling  (1952)  goodness  of  fit 
statistic  A1  =  Q  uses  the  weighting  function 

Since  the  CDF  ranges  from  zero  at  x  =  0  to  one  at  x  =  oo,  the  denominator  starts  at  zero 
increases  and  then  decreases.  Therefore,  the  weighting  function,  which  is  the  inverse  of 
the  denominator,  starts  at  infinity  decreases  and  eventually  increases  back  to  infinity.  As 
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a  result,  the  Anderson-Darling  “goodness  of  fit”  statistic  A  weights  the  tails  of  the 
distribution  much  heavier  than  the  center  of  the  distribution. 

Stephens  (1974)  presents  computational  formulas  for  these  goodness  of  fit  statistics 
that  assume  a  sample  of  random  variables  at  which  cumulative  probabilities  increases  to 
i/n,  for  i=  1,  ...«,  at  the  point  of  the  rth  random  variable  in  ascending  order.  In  contrast, 
we  have  annual  cost  reports  that  provided  the  cumulative  percent  at  the  end  of  each  year. 
In  this  application,  F„(i)  is  known  for  i=  1 ,...,«.  Following  the  derivation  of  the 
computational  formula  in  Crown  (1997),  we  develop  the  appropriate  calculation  formula 
for  this  application. 

The  Anderson-Darling  goodness  of  fit  statistic  is  defined  as 

,t=+co 

A2  =n  {[f;,  (x)  -  Fg  (x)f  [F,  (x)(l  -  F„  (*))]-'  dFo (x) 

,V=-GO 

We  define  u  =  F0  (x) .  If  the  density  function  corresponding  Fa  (x)  is  continuous  then  the 
cumulative  density  function  is  strictly  monotonically  increasing,  and  hence  invertible. 
Furthermore,  the  cumulative  density  function  is  limited  from  zero  to  one  inclusive.  We 
substitute,  and  rewrite  by  partial  fractions,  to  obtain 

A>=n-lk &atAdu 

Jo  M0~w) 

=  „■  f  (F,XK\“»f  -2F.(K'W>u  +  »2 

1  »(!-») 

2  “  (*-») 


3-20 


Expanding  the  numerator  and  completing  the  squares,  we  get 


We  note  that  the  empirical  distribution  function  only  changes  at  integer  values,  when 
the  annual  costs  are  reported.  Hence,  F„(i)  is  constant  over  the  interval  from  i  to  /+1.  At 
each  point  i,  the  function  F„(i)  has  a  jump  discontinuity.  If  we  set  the  limits  of 
integration  equal  to  the  jump  discontinuities,  we  may  determine  the  value  of  the  integral 
using  the  constant  for  that  integral.  Defining  Fo(0)=0  and  F0(n+ 1  )=  1 ,  we  obtain 
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i=n+i  T 


»z  i::™)  -c 


j=l 


r=/-;(/-l)  l«=/^(M) 


<=A/+Ir  1 

"  O'  - 1>)2  ('  n(^  (0)  - 1  n(/^0  (/  - 1)))  -  (/  - 1)  - 1)2  (lnO  -  (0)  -  ln(l  -  (/  -  1»)J 

M 

i=n+] 


=  "£[(^  (O)2  On(^  O' + 1))  - 1 n(^0  (0))  -  (/%,  (/  - 1)  - 1)2  (1  n(l  -  (0)  -  ina  -  O'  - !)))]-« 

1=1 

Law  and  Kelton  (1 991 :3 92)  present  critical  values  for  an  adjusted  Anderson-Darling  test 
statistic.  The  adjusted  test  statistic  is 


0.2 

1  + 


V  yfn ) 

For  a  Weibull  distribution,  they  give  the  90%  critical  value  as  0.637.  We  reject  the  null 
hypothesis  that  the  distribution  is  Weibull  when  the  calculated  Anderson-Darling  test 
statistic  exceeds  this  critical  value. 
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With  the  data  in  ascending  order,  the  formula  for  the  Cramer-von  Mises  is 


w'-l 


/=  1 


2/  —  1 
2n 


-|2 


+  - 


1 


12/7 


The  expected  value  of  the  Cramer-von  Misses  statistic  is  1/6,  so  some  multiply  it  by  6. 
Watson  (1961)  proposed  an  adjustment  to  the  Cramer-von  Misses  to  correct  for  the 
sample  mean.  Again,  we  calculate  the  Cramer-von  Mises  statistic  for  data  at  fixed 
intervals. 


W1  =n  jfc(^l(«))-x]2</« 


w=0 


=  ri 


u=F0(\)u=Fa(2)  h=1 


j+  }+•••  J1 Fs^'m-ujdu 


u=0  tt=F0(\)  tt=F0(n) 


,=„+]  «=/•,(<) 

nYj  J  k(^'(w))_wr du 

<=1  u=F„(i- 1) 


Let  g(u)  =  Fn(F0  ( u )  -  u ),  where  F0  (u)  is  consistent  in  any  interval [/-l,/] 

so^=-i  and  l(g(u)ydu  =  =lML+c 

du  J  ^ 


/=n+l  »=/•;.(')  . 

'=■  »=/•;,  (<-i) 

;=»+] 

=  «  X  jk(w)f  du 

'=i  «=/„(/-d 


i=i 
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=  n 


-”  +  1  _  if/  V,  ' 

/=1  $ 


»=/’o(0 
«=/'o  (M) 


»=/’0  (0 
(/■-!) 


=  f  sk(0  -  F,(i  - 1))1  -  (F0(i  - 1)  -  F„(i  - 1))1  ] 
->  /=! 


Bush  and  Moore  present  critical  values  for  the  Cramer- von  Misses  test  statistic  (Bush, 
1983:2469).  For  a  Weibull  distribution  with  fi-  2  (Rayleigh),  they  give  the  90%  critical 
values  for  various  sample  sizes.  The  critical  values  for  sample  sized  of  10,  20,  and  30  are 
0.134,0.139  and  0.141,  respectively.  The  sample  size  equals  the  number  of  budget  years 
plus  outlay  years  minus  one,  and  we  linearly  interpolate  between  the  critical  values  as 
appropriate.  We  reject  the  null  hypothesis  that  the  distribution  is  Weibull  when  the 
calculated  Cramer-von  Misses  test  statistic,  W2  exceeds  the  critical  value. 


Goodness  of  Fit  Results 

When  we  use  the  standard  service  outlay  rates,  the  initial  program  budgets  generally 
support  Rayleigh  expenditure  profiles.  The  Cramer-von  Misses  supports  59.5%  and  the 
Anderson-Darling  70.3%  of  the  programs  tested.  The  theoretical  acceptance  rate  is  90%. 
Table  3-10  shows  Rayleigh  goodness  of  fit  test  results  for  all  of  the  initial  expenditure 
profiles  tested.  In  our  regression  model,  we  include  the  test  statistics  from  the  initial 
expenditures  as  indicators  of  cost  overruns  and  schedule  slips. 
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Table  3-10  Initial  Expenditure  Profile  Goodness  of  Fit  Results 


#  Program 

Kolmogorov 

Cramer- 

von  Mises 

IrTTT - 

CvM 

Rayleigh? 

Anderson- 

Darling 

- TTTTE - 

A-D 

Rayleigh? 

■EHBH 

■Bm 

Accept 

Accept 

2  ASPJ-J 

0.2283 

0.0968 

Accept 

0.3879 

Accept 

3  B-1B 

0.1016 

0.1275 

Accept 

0.5076 

Accept 

4  Battleship 

0.0335 

0.0998 

Accept 

0.5537 

Accept 

5  IUS 

0.0995 

0.0844 

Accept 

0.4357 

Accept 

0.1434 

Tiaiaw 

Mjgaign 

Accept 

7  Kiowa 

0.3282 

0.1632 

REJECT 

0.7328 

REJECT 

8  Lantirn 

0.1445 

0.1020 

Accept 

0.4110 

Accept 

9  Trident-MSL 

0.8195 

0.1591 

REJECT 

0.6747 

REJECT 

10  Trident-Sub 

0.1853 

0.1192 

Accept 

0.8904 

REJECT 

1 1  RPV(Aquila) 

0233 

Accept 

Accept 

12  MK48  ADCAP 

0.1330 

0.0900 

Accept 

0.4418 

Accept 

13  E-6(Tacamo) 

0.4994 

0.1602 

REJECT 

0.7226 

REJECT 

14  Avenger 

0.0402 

0.1842 

REJECT 

0.9884 

REJECT 

15  PLS 

0.0934 

0.2751 

REJECT 

1.5696 

REJECT 

Accept 

0.4956 

Accept 

17  Longbow 

0.2359 

Accept 

0.4056 

Accept 

18  CMU 

Accept 

0.2808 

Accept 

19  AOE6 

0.1343 

REJECT 

0.5992 

Accept 

20  TRITAC 

0.1142 

Accept 

0.2474 

Accept 

0.6417 

07TT74 

22  JSOW 

0.3735 

0.1248 

Accept 

0.5040 

Accept 

23  ASAT 

0.6909 

0.1076 

Accept 

0.4466 

Accept 

24  ADDS 

0.0443 

0.0866 

Accept 

0.4327 

Accept 

25  LCAC 

0.4772 

0.1862 

REJECT 

0.7363 

REJECT 

26  LSD41 

Accept 

Accept 

27  MK  50 

0.0751 

Accept 

0.3183 

Accept 

28  Backscatter 

0.0753 

Accept 

0.5358 

Accept 

29  Peacekeeper 

0.1804 

REJECT 

1.0162 

REJECT 

30  T46A 

0.2109 

REJECT 

1.0077 

REJECT 

31  Cvhelo 

0.1529 

REJECT 

mMaaaamm 

REJECT 

32  TA01870iler 

0.1658 

0.4641 

REJECT 

7.7864 

REJECT 

33  FDS 

0.5654 

REJECT 

mmm 

Accept 

34  ATARS 

0.3818 

0.1134 

Accept 

Accept 

35  SRAMII 

0.5136 

0.1362 

REJECT 

Accept 

0527 

MasnariiMB 

0.6737 

reject 

37  TOW2 

0.0848 

0.1129 

Accept 

0.7955 

REJECT 

The  expenditures  based  on  final  budgets  distributed  Rayleigh  for  56.8%  based  on 
Cramer- von  Misses  and  5 1 .4%  based  on  Anderson-Darling.  We  question  our  basic 
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assumption  that  total  program  cost  for  R&D  programs  result  in  Rayleigh  expenditures, 
based  on  these  low  hypothesis  acceptance  rates.  Table  3-1 1  shows  the  Rayleigh 
goodness  of  fit  test  results  for  the  final  expenditure  profiles. 


Table  3-11.  Final  Expenditure  Profile  Goodness  of  Fit  Results 


#  Program 

Kolmogorov 

Cramer- 

von  Mises 

Pinal 

CvM 

Rayleigh? 

Anderson- 

Darling 

Final 

A-D 

Raylirgh? 

mmsmswam 

0.5O44 

REJECT 

0.6565 

REJECT 

2  ASPJ-J 

Accept 

0.4875 

Accept 

3  B-1B 

Accept 

0.5520 

Accept 

4  Battleship 

0.0295 

0.1023 

Accept 

0.5342 

Accept 

5  IUS 

0.0948 

0.1311 

Accept 

0.8506 

REJECT 

6  KC-135R 

0.4357 

0.1453 

REJECT 

0.6292 

REJECT 

7  Kiowa 

0.3049 

0.1657 

REJECT 

0.7627 

REJECT 

8  Lantirn 

0.1533 

0.1051 

Accept 

0.5050 

Accept 

9  Trident-MSL 

0.8082 

0.1780 

REJECT 

0.8057 

REJECT 

10  Trident-Sub 

0.1702 

0.1810 

REJECT 

1.8601 

REJECT 

11  RPV(Aqulla) 

0.6166 

0.1115 

Accept 

0.4702 

Accept 

12  MK48  ADCAP 

0.0639 

0.0880 

Accept 

0.6113 

Accept 

13  E-6(Tacamo) 

0.4509 

0.1467 

REJECT 

0.6786 

REJECT 

14  Avenger 

0.0469 

0.1800 

REJECT 

0.9372 

REJECT 

15  PLS 

0.0984 

0.3444 

REJECT 

2.4180 

REJECT 

16  RSlP 

(525333 

MW 

Accept 

0.4527 

Accept 

17  Longbow 

0.0822 

0.0623 

Accept 

0.2289 

Accept 

18  CMU 

0.4497 

0.0634 

Accept 

0.2642 

Accept 

19  AOE6 

0.0665 

0.1400 

REJECT 

0.8563 

REJECT 

20  TRITAC 

0.0858 

0.0538 

Accept 

0.2943 

Accept 

21  MLRS-TGW 

0.4975 

0.0873 

Accept 

0875 

Accept 

22  JSOW 

0.4943 

0.1376 

REJECT 

0.6395 

REJECT 

23  ASAT 

0.6586 

0.0801 

Accept 

0.3096 

Accept 

24  ADDS 

0.0551 

0.0639 

Accept 

0.4575 

Accept 

25  LCAC 

0.1306 

0.1025 

Accept 

0.5828 

Accept 

26  LSD41 

0.0445 

0.0980 

Accept 

0.6382 

REJECT 

27  MK  50 

0.3247 

0.0726 

Accept 

0.3457 

Accept 

28  Backscatter 

0.0428 

0.0954 

Accept 

0.6647 

REJECT 

29  Peacekeeper 

0.0371 

0.2476 

REJECT 

1.4433 

REJECT 

30  T46A 

0.7087 

0.2113 

REJECT 

1.0080 

REJECT 

31  Cvhelo 

0.0417 

O7T0 

REJECT 

REJECT 

32  TA01870iler 

0.1102 

0.5247 

REJECT 

REJECT 

33  FDS 

0.4766 

0.1012 

Accept 

0.4135 

Accept 

34  ATARS 

0.5331 

0.1580 

REJECT 

0.5654 

Accept 

35  SRAMII 

0.6240 

0.1632 

REJECT 

0.7115 

REJECT 

0533 

0.1112 

Accept 

Accept 

0.1494 

0.1128 

Accept 

REJECT 
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Regression  Analysis 


Regression  is  a  mathematical  tool  used  to  describe  a  future  response.  It  is  based 
upon  the  correlation  of  the  independent  and  dependent  variables.  Correlation 
demonstrates  a  mathematical  relationship  between  variables,  but  does  not  establish  cause 
and  effect  relationships. 

This  regression  analysis  includes  two  types  of  independent  variables,  quantitative 
and  qualitative  (McClave  et  al.,  1998:579).  Quantitative  variables  are  measured  on  a 
continuous  numerical  scale.  Variables  that  represent  categorical  values  are  qualitative. 
We  use  qualitative  variables  to  express  a  situation  expressed  in  discrete  terms.  For 
example  “Weibull  Location  Indicator”  is  a  qualitative  variable  since  it  is  a  binary  variable 
that  describes  whether  the  “Weibull  Location”  is  greater  than  zero  or  not. 

This  study  uses  regression  to  establish  a  relationship  between  the  independent 
variables  collected  through  the  cost  model  parameter  estimation  and  goodness  of  fit 
processes  and  the  dependent  variables  of  cost  growth  and  schedule  growth.  We  fit  the 
data  to  the  linear  model  using  a  least  squares  approach. 

Full  Regression  Model.  The  first  step  in  creating  the  model  is  to  determine 
relationships  between  the  dependent  variables  and  the  independent  variables.  We 
identify  the  relationship  graphically  by  plotting  each  of  the  independent  variables  against 
the  dependent  variable.  We  identify  an  approximate  mathematical  relationship  by 
looking  for  general  trends  in  the  plots.  The  plots  of  our  data  in  Appendix  B  identify 
possible  linear  relationships  for  certain  variables,  but  no  discemable  higher  order 
relationships. 
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The  full  model  includes  all  possible  combinations  of  variables,  including 
categorical  and  interaction  terms.  The  full  model  equation 

y  =  fio  +  P1X1  +  P2X2 . +  P&i  +  s 

where  y  is  the  dependent  variable,  the  Pi  represent  parameters  with  unknown  values  and 
the  X,  represent  independent  variables.  The  x,  may  represent  higher  order  or  interaction 
terms  such  as  xp,  or  x^x,,. 

Since  the  Weibull  and  Rayleigh  parameters  describe  similar  models  to  the  same 
dataset,  a  high  degree  of  correlation  exists  between  them.  To  avoid  multicollinearity,  a 
phenomenon  that  indicates  multiple  variables  are  explaining  the  same  phenomena,  we 
separate  the  Rayleigh  parameters  and  Weibull  parameters  into  different  full  regression 
models.  Therefore,  we  evaluate  four  full  models:  Cost  growth  with  Weibull  parameters, 
cost  growth  with  Rayleigh  parameters,  schedule  growth  with  Weibull  parameters,  and 
schedule  growth  with  Rayleigh  parameters.  In  Table  3-12,  we  list  the  variables  contained 
in  our  full  models. 


Table  3-12.  Regression  Full  Model  Variables 


Cost  Variables  -  Set  1 

Cost  Variables  -  Set  2 

S  Cost  Overrun  (dependent) 

Yrs  Initial  SAR 

Weibull  Cost  Factor 

Weibull  Scale 

Weibull  Shape 

Weibull  Location 

Kolmogorov 

Cramer-von  Mises 

Anderson -Darling 

Interaction  Weibull  Shape  /  Weibull  Scale 

S  Cost  Overrun  (dependent) 

Yrs  Initial  SAR 

Rayleigh  Cost  Factor 

Rayleigh  Location 

Rayleigh  Scale 

Kolmogorov 

Cramer-von  Mises 

Anderson-Darling 

Scheduel  Variables  -  Set  1 

Schedule  Variables  -  Sei  2 

%  Schedule  Overrun  (dependent) 

Yrs  Initial  SAR 

Weibull  Cost  Factor 

Weibull  Scale 

Weibull  Shape 

Weibull  Location 

Kolmogorov 

Cramer-von  Mises 

Anderson-Darling 

Interaction  Weibull  Shape  /  Weibull  Scale 

S  Cost  Overrun  (dependent) 

Yrs  Initial  SAR 

Rayleigh  Cost  Factor 

Rayleigh  Location 

Rayleigh  Scale 

Kolmogorov 

Cramer-von  Mises 

Anderson-Darling 
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Model  Reduction.  Model  reduction  is  the  process  of  eliminating  variables  that  do 


not  add  explanatory  capability  to  the  model.  We  systematically  remove  variables  from 
the  full  model  based  on  their  relative  contribution  to  the  regression  model.  This 
reduction  iteratively  removes  the  least  significant  variables,  until  the  overall  significance 
of  the  model  is  reduced.  We  test  the  each  reduced  model  to  determine  if  the  reduced 
model  is  statistically  equivalent  to  the  model  before  the  reduction. 

We  select  variables  for  removal  by  examining  their  observed  significance  level  or 
p- value.  Statistical  programs,  like  JMP™,  provide  a  p-value  for  each  variable  that  may 
be  used  to  make  this  evaluation.  If  the  p-value  is  greater  than  desired  significance  level 
a,  the  variable  may  be  removed  and  the  newly  reduced  model  tested.  (McClave  et  al., 
1998:  452-453).  We  reduce  the  full  model,  until  the  reduced  model  is  not  statistically 
equivalent  to  the  model  before  reduction. 

From  the  full  models  in  Table  3-12  we  iteratively  reduced  the  number  of 
variables,  by  evaluating  each  variable’s  contribution  to  the  model.  We  considered  each 
variable’s  p-value,  an  indicator  of  the  explanatory  value  of  the  variable.  Generally,  we 
eliminated  variables  with  a  p-value  greater  than  0.05,  provided  their  removal  does  not 
adversely  affect  the  overall  statistics  of  the  model,  as  determined  by  an  F-test. 

We  use  the  F-test  to  determine  if  the  reduced  model  is  statistically  equivalent  to 
the  model  before  reduction.  The  test  statistic  for  the  F-test  is 

„  (■ SSEr-SSE,)l(P,-l)r ) 

— 

where  SSEr  is  the  reduced  model  sum  of  squared  errors,  SSE/  is  the  sum  of  squared  errors 
for  the  model  before  reduction,  (if  is  the  number  of  parameters  in  the  model  before 
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reduction,  (3r  is  the  number  of  parameters  in  the  reduced  model,  and  dff  is  the  degrees  of 
freedom  for  the  full  model  before  reduction.  The  rejection  region  for  this  test  statistic  is 
F>  Fa.  In  other  words,  if  F  is  less  than  Fa ,  the  reduced  model  is  statistically  as  good  as 
the  model  before  reduction.  Fa  is  based  on  v/  =  (J3f-  /3r)  numerator  degree  of  freedom 
(< df)  and  =  ^denominator  df  (McClave  et  al.,  1998:  600).  We  chose  a  to  be  0.05,  for 

95%  confidence  in  our  models. 

After  creating  the  model,  we  evaluated  how  well  the  model  describes  the 
relationship  between  dependent  and  independent  variables  using  the  coefficient  of 
multiple  determination,  Rr.  However,  R  alone  can  be  misleading,  since  R  depends  on 
the  model,  sample  size  and  sample.  Adding  more  independent  variables  to  the  model 
always  increases  R2.  The  measure  of  adjusted  R2  accounts  for  the  number  of  explanatory 
variables  relative  to  the  number  of  observations.  Therefore,  we  use  adjusted  R2  generally 
for  comparison  between  statistical  models,  while  we  use  R2  to  evaluate  the  explanatory 
capacity  of  a  given  model.  While  we  prefer  to  have  R2  as  close  to  1 .0  as  possible,  using 
real  world  data  can  result  in  relatively  low  R2. 

Arriving  at  the  reduced  form  of  the  model,  we  evaluate  how  well  it  subscribed  to 
the  basic  error  term  assumptions  of  normality,  constant  variance  and  independence.  Each 
of  the  models  essentially  passed  the  tests  for  the  basic  assumptions.  We  detail  these 
results  in  the  model  discussion  section. 

We  also  evaluated  each  of  the  final  models  for  outliers.  Cook's  distance  or 
Cook’s  D  Influence  measures  the  overall  influence  of  each  observation  on  the  regression 
coefficients,  including  the  intercept.  Cook  proposes  that  the  influence  of  the  /th  data 
point  be  measured  by  the  distance  D,  where 
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D,  =  {b  -  b(/)}'  X’  X{b  -  b(z)}/(/?s2) 

where  X  is  n  x  p,  b  is  the  usual  least  squares  estimator,  s 2  is  the  variance  of  the  error,  and 
b(z)  is  the  least  squares  estimator  obtained  after  the  /th  data  point  is  removed.  Cook’s  D 
provides  useful  information  in  identifying  outliers,  since  it  checks  whether  the  deletion  of 
one  or  more  critical  observations  greatly  affects  the  model  (Draper,  1 98 1 : 1 70).  Data  that 
have  a  significant  influence  on  the  overall  model  may  be  candidates  for  removal  as 
outliers.  On  the  several  occasions  when  we  identified  an  outlier,  we  removed  the  data 
point  and  recalculated  the  regression  model. 

The  dependent  variable  of  cost  growth  is  the  actual  cost  overrun  in  constant  year 
2000  dollars.  For  example,  the  RSIP  program  initially  budgeted  $449.8  CY$00M  and  the 
final  program  showed  $477.4  million  CY$00M,  a  cost  growth  is  $27.6  CY$00M.  All 
cost  numbers  in  the  dataset  are  in  millions  of  constant  year  2000  dollars.  A  negative  cost 
growth  indicates  that  the  final  program  cost  less  than  originally  planned.  The  37 
programs  show  a  range  of  cost  growth  from  -$688.6M  to  $1 199.3M,  with  an  average  cost 
growth  of  $40. 3M.  The  final  cost  model  is  based  on  a  dataset  with  B-1B  data  removed. 
We  removed  the  Bl-B  data  because  the  residual  plot  indicates  it  as  an  extreme  outlier 
that  significantly  influences  the  model  parameter  estimates.  The  next  highest  cost  growth 
to  the  Bl-B’s  $1 199.3  was  the  AAMRAAM-J’s  at  $500.9  (see  Table  3-2). 

Final  Cost  Model.  The  cost  growth  model  is 

$  Cost  overrun  =  -0.0558  Cost  of  Initial  SAR  +  23.92  Weibull  Scale  -  59.36 
The  cost  model  sensitivity  shown  in  Table  3-13  indicates  the  total  contribution  of  each 
variable  to  the  predicted  cost  growth.  Our  model  indicates  that  the  Weibull  Scale 


3-31 


parameter  has  the  most  significant  contribution  towards  cost  growth  on  average.  The 
scale  parameter  is  offset  by  the  contribution  of  the  initial  budget  variable. 


Table  3-13.  Cost  Model  Sensitivity 


Variable 

- CoeTT - 

Inputs 

Outputs 

Cost  Initial  SaR 

'4r>-  vVHiKEEUttsKL* 

Min 

TTM2 

-0.8635787 

Max 

13686.313 

-763.3951927 

Avg 

1252.450 

-69.8591463 

Med 

521.738 

-29.1014807 

Weibull  Scale 

23.920513 

Min 

1.056 

25.2610278 

Max 

13.952 

333.7271738 

Avg 

5.760 

137.7898914 

Med 

5.295 

126.6650761 

Figure  3-6  shows  the  statistics  for  the  final  cost  model.  Each  of  the  variables  is 
significant;  the  p-value  in  the  Parameter  Estimate  section  is  less  than  0.05.  The  model  is 
significant  since  the  F-ratio  probability  has  a  value  less  than  0.05.  The  model  explains 
53.4%  of  the  total  cost  growth  variation  in  the  dataset. 


Response:  $  Cost  CVerrun 


Summary  of  Fit 

RSquare  0.533828 

RSquare  Adj  0.505576 

Root  Mian  Square  Error  155.0953 

Mean  of  Response  8.132778 

Observations  (or  Sum  Wgts)  36 


Parameter  Estimates 

Term  Estimate  Std  Error  t  Ratio  FVob>jt| 

Intercept  -59.35861  57.61211  -1.03  0.3104 

Cost  Initial  -0.055778  0.009771  -5  71  <  0001 

W  scale  23.920513  8.869031  2.70  0.0109 

Effect  Test 

Source  Nparm  DF  Sum  of  Squares  F  Ratio  Prob>F 

Cost  Initial  1  1  783842.68  32.5860  <  0001 

W  scale  1  1  174979.07  7.2743  0.0109 


Wrote- Model  Test 


1  Andysis  of  Variance  I  | 

Source  DF  Sum  of  Squares  Mean  Square  F  Ratio  ■ 

Model  2  909007.2  454504  18.8947  j 

Error  33  793800.6  24055  Prob>F  j  ' 

C  Total  35  1702807.9  <.0001  I  j 
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Figure  3-6.  Final  Cost  Model  Statistics 


Quantiles 

1  Moments  ; 

maximum 

100.0% 

429.34 

Mean 

-0.0000 

99.5% 

429.34 

Std  Dev 

150.5989 

97.5% 

429.34 

Std  Error  Mean 

25.0998 

90.0% 

156.08 

Upper  95%  Mean 

50.9550 

quartile 

75.0% 

31.61 

Lower  95%  Mean 

-50.9550 

median 

50.0% 

-12.98 

N 

36.0000 

quartile 

25.0% 

-39.28 

Sum  Weights 

36.0000 

10.0% 

-138.80 

— 

2.5% 

-382.11 

0.5% 

-382.11 

minimum 

0.0% 

-382.11  i 

Figure  3-7.  Final  Cost  Model  Residual  Analysis 


Quantiles 

Moments 

Test  for  Normality 

maximum 

100.0% 

2.8400 

Mean 

-0.00339 

Shapiro-Wilk  W  Test 

99.5% 

2.8400 

:  Std  Dev 

0.99568 

W  Prob<W 

97.5% 

2.8400 

Std  Error  Mean 

0.16595 

0.859862  0.0002 

90.0% 

1.0354 

;  Upper  95%  Mean 

0.33350 

quartile 

75.0% 

0.2148 

:  Lower  95%  Mean 

-0.34028 

median 

50.0% 

-0.1136 

N 

36.00000 

quartile 

25.0% 

-0.2752 

Sum  Weights 

36.00000 

10.0% 

-0.9327 

— 

- - 

2.5% 

-2  5285 

0.5% 

-2.5285 

minimum 

0.0% 

-2.5285 

Figure  3-8.  Final  Cost  Model  Studentized  Residual  Analysis 


Figures  3-7  and  3-8  show  the  cost  model  residual  plot  and  Studentized  residual 
plot,  respectively.  The  intent  of  these  plots  is  to  indicate  whether  the  model  satisfies  the 
assumption  that  the  residuals  are  normally  distributed.  As  indicated  by  Figures  3-7  and 
3-8,  the  residuals  for  the  cost  model  are  too  heavy  in  the  tails  of  the  distribution  to  be 
considered  normal.  In  other  words,  the  presence  of  the  relatively  tall  bars  near  three 
standard  deviations  indicates  that  the  residuals  may  not  be  normally  distributed.  On  the 
other  hand,  Figures  3-7  and  3-8  indicate  the  residuals  are  not  highly  skewed;  they  are 
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fairly  symmetric.  The  cost  model,  however,  fails  the  requirement  for  normality  by 
inspection  of  the  residual  plots  and  also  according  to  the  Shapiro-Wilk  test.  If  the 
Shapiro-Wilk  p-value  reported,  shown  as  Prob<W,  is  less  than  .05  (or  some  other  alpha), 
then  you  conclude  that  the  distribution  is  not  normal.  However,  regression  is  robust  with 
respect  to  this  condition,  so  the  model  is  still  acceptable.  Normality  primarily  drives 
confidence  intervals.  Since  we  do  not  make  predictions  with  confidence  intervals,  we 
accept  the  model. 


Cook>E$  D  Influences  Cost  overru  j 

Figure  3-9.  Final  Cost  Model  Cook’s  D  Plot 

The  Cook’s  D  test  for  influential  points  reveals  that  no  data  in  the  final  model 
significantly  influence  the  results.  Generally,  we  consider  a  point  with  a  Cook’s  D  value 
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greater  than  1.0  to  be  influential.  Figure  3-9  shows  that  none  of  the  data  in  the  final 
model  significantly  influence  the  model. 

Final  Schedule  Model.  The  final  schedule  model  is  based  on  a  dataset  with 
Trident  Submarine  removed.  We  removed  the  Trident  Submarine  program  because  it  had 
a  schedule  slip  of  3.2,  a  clear  outlier  since  no  other  slips  were  nearly  as  large,  as  seen  in 
Table  3-2.  The  schedule  growth  model  is  the  following: 

%  Schedule  Growth  =  0.00003  Initial  Budget  -  0.047  Weibull  Scale 
+  0.22  Weibull  Shape  -  2.25  Interaction  Weibull  Shape  /  Scale 
-  1.24  Kolmolgorov  +  8.69  Cramer  Von  Mises 
-  0.45  Anderson-Darling  +  1 .54 

The  schedule  model  sensitivity  in  Table  3-14  indicates  the  total  contribution  of 
each  variable  to  the  predicted  schedule  growth.  Our  model  indicates  that  the  Weibull 
Shape  and  Cramer-von  Mises  variables  have  the  most  significant  contribution  towards 
positive  cost  growth  on  average.  The  interaction  term  has  the  greatest  offset  contribution. 
The  positive  schedule  growth  represented  by  Cramer-von  Mises  variable  is  offset  by  both 
the  Kolmolgorov  and  Anderson-Darling  variable  contributions.  This  model  explains 
50.5%  of  the  variation  in  schedule  growth  for  the  36  included  programs. 
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Table  3-14.  Schedule  Model  Sensitivity 


Variable 

Coeff 

Inputs 

Outputs 

Cost  Initial  SAR 

0.0000625 

Min 

\5mi 

0.0000387 

Max 

13686.313 

0.0342158 

Avg 

1395.598 

0.0034890 

Med 

576.703 

0.0014418 

Weibull  Shape 

0.2207517 

Min 

1.089 

0.2404470 

Max 

4.582 

1.0114757 

Avg 

2.666 

0.5884917 

Med 

2.473 

0.5459822 

Weibull  Scale 

-0.047334 

Min 

1.572 

-0.0743941 

Max 

13.952 

-0.6603806 

Avg 

5.930 

-0.2806735 

Med 

5.317 

-0.2516685 

Interaction  Shape  /  Scale 

-2.24977 

Min 

0.241 

-0.5417110 

Max 

0.825 

-1.8568785 

Avg 

0.492 

-1.1074555 

Med 

0.509 

-1.1445318 

KS 

-1.236105 

Min 

0.031 

-0.0389020 

Max 

0.821 

-1.0147159 

Avg 

0.310 

-0.3826174 

Med 

0.237 

-0.2928824 

CVM 

8.69238 

Min 

0.050 

0.4356635 

Max 

0.272 

2.3654553 

Avg 

0.129 

1.1186659 

Med 

0.116 

1.0116808 

AD 

-0.451847 

Min 

0.245 

-0.1108735 

Max 

1.557 

-0.7036010 

Avg 

0.599 

-0.2707557 

Med 

0.542 

-0.2447851 
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Response:  Sched  Slip 


j  i  Summary  of  Fit 


RSquare 

RSquare  Adj 

Root  Mean  Square  Error 

Mean  of  Response 

Observations  (or  Sum  Wgts) 

0.504712 

0.38089  1 

0.212852 

1.204167 

36 

:  Parameter  Estimates 

Term 

Estimate 

Std  Error  t  Ratio 

Prob>|t| 

Intercept 

1.5149604 

0.288802  5.25 

<.0001 

Cost  Initial 

0.000025 

0.000014  1.76 

0.0888 

W  scale 

-0.047334 

0.039075  -1.21 

0.2359 

W  Shape 

0.2207517 

0.126301  1.75 

0.0915 

W  Interaction 

-2.24977 

0.804842  -2.80 

0.0093 

Kol 

-1.236105 

0.343533  -3.60 

0.0012 

CVM 

8.69238 

2.51408  3.46 

0  0018 

AD 

-0.451847 

0.118422  -3.82 

0.0007 

Effect  Test 

Source 

Npamn  DF 

Sum  of  Squares 

F  Ratio 

Prob>F 

Cost  Initial 

1  1 

0.14087533 

3.1094 

0.0888 

'  W  scale 

1  1 

0.06648206 

1.4674 

0.2359 

W Shape 

1  1 

0.13840519 

3.0549 

0.0915 

W  Interaction 

1  1 

0.35400653 

7.8137 

0.0093 

Kol 

1  1 

0.58658455 

12.9472 

0.0012 

;  CVM 

1  1 

0.54159610 

11.9542 

0.0018 

.  AD 

1  1 

0.65958637 

14.5585 

0.0007 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

7 

1.2927057 

0.184672 

4.0761 

Error 

28 

1.2685693 

0.045306 

Prob>F 

C  Total 

35 

2.5612750 

0.0034 

Figure  3-10.  Final  Schedule  Model  Statistics 


Figure  3-10  shows  the  statistics  for  the  final  schedule  model.  The  inclusion  of 
the  interaction  term  “Weibull  Shape/Weibull  Scale”  necessitates  the  inclusion  of  both  the 
individual  Weibull  Shape  and  Weibull  Scale  parameters,  although  neither  is  significant  to 
the  0.05  level.  We  included  the  “Cost  Initial”  variable  because  it  affected  the  explanatory 
ability  of  the  model  significantly.  All  other  variable  have  p-values  less  than  0.05,  as 
shown  in  the  Parameter  Estimate  Section.  The  model  is  significant  since  the  F-ratio 
probability  has  a  value  less  than  0.05.  The  model  explains  50.5%  of  the  total  variation  of 
the  schedule  growth. 
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!  Residual  Sched  Slip 


-  • 

■■ 

1  1  1  1  1  1  I 

-0.6  -0.4  -0.2  -0.0  .2  .4  .6 

Quantiles 


maximum 

100.0% 

0.57517 

99.5% 

0.57517 

97.5% 

0.57517 

90.0% 

0.24160 

quartile 

75.0% 

0.10257 

median 

50.0% 

-0.0045 

quartile 

25.0% 

-0.1544 

10  0% 

-0.1826 

2.5% 

-0.4299 

0.5% 

-0.4299 

minimum 

0.0% 

-0.4299 

!  Moments  j 

Mean 

-0.00000 

1  Std  Dev 

0.19038 

Std  Error  Mean 

0.03173 

Upper  95%  Mean 

0.06442 

Lower  95%  Mean 

-0.06442 

N 

36.00000 

Sum  Weights 

36.00000 

Figure  3-11.  Final  Schedule  Model  Residual  Analysis 


Studentized  Resid  Sched  Slip 


Quantiles 

Moments 

!  Test  for  Normality 

maximum 

100.0% 

2.8801 

Mean 

0.01326 

;  Shapiro-Wilk  W  Test 

99.5% 

2.8801 

Std  Dev 

0.98118 

W  Prob<W 

97.5% 

2.8801 

Std  Error  Mean 

0.16353 

0.950232  0.1406 

90.0% 

1.1974 

Upper  95%  Mean 

0.34525 

.  . 

quartile 

75.0% 

0.5735 

Lower  95%  Mean 

-0.31872 

median 

50.0% 

-0.0234 

N 

36.00000 

quartile 

25.0% 

-0.7876 

Sum  Weights 

36.00000 

10.0% 

-0.9237 

— 

2.5% 

-2.2213 

0.5% 

-2.2213 

minimum 

0.0% 

-2.2213 

Figure  3-12.  Final  Schedule  Model  Studentized  Residual  Analysis 


Figures  3-1 1  and  3-12  show  the  schedule  model  residual  plot  and  Studentized 
residual  plot,  respectively.  The  intent  of  these  plots  is  to  indicate  whether  the  model 
satisfies  the  assumption  that  the  residuals  are  normally  distributed.  Figures  3-7  and  3-8 
indicate  that  the  residuals  indicate  are  fairly  symmetric  and  appear  to  be  normally 
distributed.  The  schedule  model  passes  the  Shapiro-Wilk  test  for  normality  with  a 
Prob<W  of  0.1406.  Therefore,  the  schedule  model  passes  the  normality  assumption. 
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Figure  3-13.  Final  Schedule  Model  Cook’s  D  Plot 


The  Cook’s  D  test  for  influential  points  reveals  that  no  data  in  the  final  model  are 
significantly  influencing  the  results.  Generally,  we  consider  a  point  with  a  Cook’s  D 
value  greater  than  1 .0  to  be  influential.  Figure  3-9  shows  that  none  of  the  data  in  the  final 
model  significantly  influence  the  model. 

Regression  Summary.  Belcher  and  Dukovich  identified  12  factors  in  three  areas 
contributing  to  development  program  cost  growth  and  schedule  growth.  (See  Figure  2-7.) 
Our  cost  and  schedule  models  each  account  for  only  a  single  of  Belcher  and  Dukovich’s 
factors,  funding  constraints.  Yet,  these  models  explain  53.4%  of  the  variation  in  cost 
growth  and  50.5%  of  the  variation  in  schedule  growth.  Appendix  B  contains  a  summary 
of  all  of  the  results  for  each  of  the  37  programs. 
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Model  Validation.  We  use  validation  to  evaluate  the  model  accuracy.  Validation 


uses  program  information  outside  the  data  used  to  create  the  model  to  test  the  model’s 
descriptive  ability.  We  excluded  each  of  the  programs  in  Table  3-15  from  the  original 
dataset  because  they  are  still  ongoing  programs  as  of  the  date  of  our  SAR  data.  Based  on 
the  estimated  completion  date  and  cost,  we  determined  that  each  of  the  programs  is 
greater  than  80%  complete,  making  them  useful  for  model  validation.  We  used  estimated 
completion  cost  for  the  six  programs  shown  in  Table  3-15  to  validate  the  model. 

Table  3-15.  Near  Completed  Programs  for  Validation 


Program 

Cost 

Initial 

1 

Weibull 

Scale 

Weibull 

Shape 

Weibull 

Location 

Weibull 

Interaction 

Kol 

CVM 

AD 

JT1DS-J 

1741.92 

21 

VTW 

J3T 

1.61 

0.6856 

■roEniaa 

DSCSIII 

328.26 

15 

1.92 

1.56 

0.54 

0.1114 

0.2905 

1.6495 

SADARM 

812.57 

9 

4.96 

2.42 

0.49 

0.1162 

0.4045 

DMSP 

685.88 

17 

6.60 

1.49 

1.61 

0.23 

0.0479 

0.0637 

0.4321 

T45TS 

784.28 

12 

6.46 

3.32 

2.16 

0.51 

0.7167 

0.1467 

0.5941 

WISWAM 

887.23 

8 

6.38 

3.47 

0.54 

We  used  a  statistical  technique  to  calculate  the  variance  of  the  new  observation  Yh(new) 
corresponding  to  X/,,  the  specified  values  of  the  X  variables.  The  variance  formula  is 

S2(Yh(new)  =  MSE(  1  +X'h(X'  X)’ 1  Xh 

where  X  is  the  matrix  of  program  input  variables,  X'  is  its  transpose,  Xh  is  the  vector 
containing  the  values  for  the  new  data  point  variables,  X'h  is  its  transpose,  and  MSE  is  the 
Mean  Squared  Error  of  the  original  regression. 

Cost  Model  Validation.  From  the  six  programs  shown  in  Table  3-15,  we 
calculated  cost  growth  using  the  cost  model.  Each  of  the  six  programs  is  within  the  range 
of  data  used  to  construct  the  regression  model.  Table  3-16  shows  the  comparison  of 
actual  to  predicted  cost  growth.  Table  3-16  also  shows  the  variance  associated  with  the 
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regression  prediction,  along  with  “z”  which  represents  the  normalized  value.  The  model 
appears  to  predict  too  little  cost  growth  in  four  of  six  cases,  with  an  overall  average  of 
$87.38  CY$00M.  The  bias  appears  to  be  about  0.56  standard  deviations. 

Table  3-16.  Cost  Model  Validation 


Program 

Actual  Cost 
growth 

Predicted 

Cost 

growth 

Actual 

Minus 

Predicted 

Variance 

z 

JTIDS-J 

-166.94 

143.72 

-310.66 

168.57 

-1.843 

DSCSIII 

547.71 

-31.89 

579.60 

160.98 

3.600 

SAD ARM 

-333.08 

-346.78 

157.44 

-2.203 

DMSP 

93.76 

46.48 

47.28 

157.41 

0.300 

T45TS 

338.65 

49.18 

289.47 

157.42 

1.839 

WISWAM 

42.90 

265.37 

157.38 

1.686 

Schedule  Model  Validation.  From  the  six  programs  shown  in  Table  3-15,  we 
calculated  schedule  growth  using  the  schedule  model.  Three  of  the  six  programs  are 
within  the  range  of  data  used  to  construct  the  regression  model.  At  least  one  variable  of 
the  JTIDS,  DSCSIII,  and  DMSP  programs  falls  outside  the  range  of  the  data  used  to 
construct  the  model.  Table  3-17  shows  the  comparison  of  actual  to  predicted  schedule 
growth,  the  variance  associated  with  the  regression  prediction,  and  the  normalized  value. 
The  model  appears  to  predict  too  little  schedule  growth  in  five  of  six  cases,  with  an 
overall  average  of  0.230.  The  bias  appears  to  be  about  0.30  standard  deviations. 


Table  3-17.  Schedule  Model  Validation 


. Actual 

Sched 

growth 

Pred  Sched 
growth 

"7\etuar  - 

Minus 

Predicted 

Variance 

z 

1.24 

— OT — 

HP 

0.TJ5 

1.47 

2.08 

-0.609 

0.385 

-1.582 

1.89 

1.39 

0.498 

0.488 

1.019 

1.41 

1.34 

0.074 

0.835 

0.089 

1.50 

0.92 

0.580 

0.531 

1.094 

1.63 

1.12 

0.509 

0.540 

0.943 

Model  Confirmation  of  Theory 

We  hypothesize  that  Rayleigh  budget  profiles  perform  better  than  the  profiles 
actually  used.  In  order  to  test  this  hypothesis,  we  created  Rayleigh  efficient  expenditure 
profiles  using  the  method  described  by  Lee,  Hogue  and  Gallagher  (1997).  Given  the  final 
costs  from  the  37  initial  programs,  we  create  a  Rayleigh  expenditure  profile. 

We  then  applied  the  procedures  outlined  earlier  in  this  chapter  to  identify  Weibull 
parameters  and  goodness  of  fit  associated  with  the  Rayleigh  expenditure  profile.  Table  3- 


1 8  identifies  the  parameters  for  the  Rayleigh  expenditures.  Using  the  resulting 
parameters  as  input  variables,  we  compare  the  expected  performance  of  the  Rayleigh 
expenditures  to  the  actual  program  performance. 

Table  3-19  shows  the  results  of  the  comparison  for  the  cost  model.  In  seven  of 


the  37  programs,  the  Rayleigh  expenditures  resulted  in  lower  predicted  cost  overruns.  In 
the  thirty  other  programs,  the  Rayleigh  expenditures  resulted  in  greater  cost  growth. 
Table  3-20  shows  the  results  of  the  comparison  for  the  schedule  model.  In  five  of  the  37 
programs,  the  Rayleigh  expenditures  resulted  in  lower  predicted  schedule  overruns.  In 
the  thirty-two  other  programs,  the  Rayleigh  expenditures  resulted  in  greater  schedule 
growth. 
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Table  3-18.  Rayleigh  Efficient  Profiles 


icient  Ironies 
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w 

Shape / 
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Shape 

Scale 
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gorov  Mises 
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32.2 

1230.7 
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32  TA01870iler 
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34  ATARS 

35  SRAM1I 


37  TOW2 
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rnriltl 


riMii 


2 
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0.21 

0.00 

0 

0.00 

0.022 

0.123 

2 

6.95 

0.29 

0.00 

1 

0.00 

0.043 

0.232 

2 

4.28 

0.47 

0.00 

0 

0.00 

0.111 

0.592 

2 

5.35 

0.37 

0.00 

0 

0.00 
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MU 


nilttl 


0  0.00  0.037 
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2 
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0.27 

0.00 

1 

0.00 
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0.201 

2 
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0 

0.00 

0.012 
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2 
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0.27 

0.00 

1 

0.00 
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0.201 

2 

6.41 

0.31 

0.00 

1 

0.00 

0.050 

0.271 
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Table  3-19.  Cost  Model  Confirmation  Results 


I 


Table  3-20.  Schedule  Model  Confirmation  Results 


We  hypothesize  that  a  Rayleigh  budget  profile  performs  better  than  the  profile 
actually  used.  However,  this  hypothesis  proved  incorrect,  as  the  majority  of  programs 
performed  worse  with  a  Rayleigh  derived  budget  profile.  This  leads  to  the  important 
question  of  what  expenditure  profile  parameters  provide  the  most  efficient  program 
execution,  in  terms  of  program  growth. 

Chapter  Summary 

This  chapter  describes  the  methodology  and  results  associated  with  testing  for  a 
relationship  between  an  initial  budget  profile  and  cost  and  schedule  growth.  We  detail 
the  procedure  for  converting  budgets  to  outlays  and  describe  the  technique  for  estimating 
the  Weibull  and  Rayleigh  parameters  for  the  outlays.  We  evaluate  if  the  expenditures  are 
Rayleigh  using  goodness  of  fit  statistics.  We  test  for  a  relationship  between  the  above 
parameters  and  statistics  and  program  growth  with  regression.  Finally,  we  test  the 
hypothesis  that  Rayleigh  budget  profiles  perform  better  than  the  actual  budget  profiles. 
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IV.  Conclusion  and  Recommendations 


Chapter  Overview 

We  present  summary  finding  and  conclusions  in  this  chapter.  We  discuss  the 
general  conclusions  of  this  research,  including  overall  model  performance  and 
limitations.  We  also  discuss  the  potential  of  future  research,  based  on  our  findings. 

Conclusion 

We  conclude  that  a  statistically  significant  relationship  exists  between  initial 
budget  profile  and  cost  and  schedule  growth.  Based  on  the  results  of  the  regression 
models,  we  explain  over  fifty  percent  of  the  variation  for  both  cost  and  schedule  growth. 
Furthermore,  the  results  of  our  model  validation  indicate  that  both  the  cost  and  schedule 
models  perform  acceptably.  Five  of  the  six  test  programs  for  cost  and  all  six  of  the  test 
programs  for  schedule  resulted  in  less  than  three  standard  deviations  difference  from  the 
expected  result.  Also,  we  observe  from  the  validation  results  that  both  the  cost  model 
and  schedule  model  tend  to  underestimate  program  growth.  The  cost  model  tends  to 
underestimate  by  an  average  of  $87.4M  CY$00,  while  the  schedule  model  is  low  by  an 
average  of  0.30  standard  deviations. 

The  main  model  limitation  derives  from  the  type  of  data  used  to  create  the  model. 
The  SAR  database  presents  budget  data  in  a  variety  of  formats,  severely  limiting  the 
amount  of  useable  data.  Also,  the  data  included  in  the  model  include  total  program 
budgets,  a  subset  of  the  contract  data  used  in  previous  studies.  While  using  program 
budget  data  increases  the  usefulness  and  robustness  of  the  model,  top  level  data  tends  to 
be  noisy.  Finally,  according  to  the  Dukovich  and  Belcher  development  cost  model,  we 
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examined  only  one  of  the  many  potential  contributors  to  development  cost.  However,  we 
explained  53.4%  of  the  variation  in  cost  growth  and  50.5%  of  the  variation  in  schedule 
growth.  Ideally,  we  would  like  to  be  able  to  include  more  of  the  contributors  to  increase 
the  accuracy  of  the  models. 

Recommendation 

The  Rayleigh  expenditure  profile  does  not  appear  to  be  the  best,  based  on  these 
regression  models.  However,  this  fact  leads  to  the  question  of  what  parameter  choices 
might  lead  to  better  performance  than  the  actual  performance.  Future  thesis  efforts  may 
identify  the  Weibull  parameters  that  provide  the  most  efficient  program  execution,  in 
terms  of  program  growth. 
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Appendix  A.  Service  Inflation  Indices 


Appendix  Overview 

This  appendix  provides  FYOO  raw  and  weighted  inflation  indices  for  the  Air  Force, 
Army,  and  Navy. 


A-l 


Table  A-2.  Army  Inflation  Indices 
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Table  A-3.  Navy  Inflation  Indices 

NavyFYOO  Inflation  Rates,  RDT&BN 


FY 

1970 

iw 

1972 

1973 

1974 

1975 

1975 

1977 

1978 

1979 

Raw 

0.2625 

0.2760 

0.2887 

0.3012 

0.3253 

0.3608 

0.3847 

0.4060 

0.4336 

0.4700 

Weighted 

0.2706 

0.2831 

0.2959 

0.3133 

0.3406 

0.3706 

0.3923 

0.4198 

0.4520 

0.4992 

FY 

1980 

1981 . 

1982 

1983 

1984 

1985 

1986 

1987 

1988 

1989 

Raw 

0.5198 

0.5749 

0.6186 

0.6489 

0.6736 

0.6965 

0.7160 

0.7353 

0.7574 

0.7892 

Weighted 

0.5520 

0.6021 

0.6339 

0.6624 

0.6863 

0.7076 

0.7281 

0.7496 

0.7747 

0.8073 

FY 

1990 

1991 

1992' 

“TOT 

1994 

1995 

1996 

1997 

'  1998 . 

""T999 . 

Raw 

0.8208 

0.8561 

0.8800 

0.9038 

0.9219 

0.9394 

0.9582" 

0:9754“ 

0.9822 

0.9901 

Weighted 

0.8403 

0.8705 

0.8958 

0.9165 

0.9337 

0.9517 

0.9677 

0.9795 

0.9874 

0.9966 

FY 

2000 

2001 

2002 . 

2003 

2004 

20o5 

2006 

2007 

2008 

2009 

Raw 

1.0000 

1.0150 

. 1.0302 . 

1.0457 

1 .0666 

1.0879 

1.1097 

1.1319 

. 1.1545 . 

1.1776'" 

Weighted 

1.0090 

1.0243 

1.0400 

1.0580 

1.0792 

1.1008 

1.1228 

1.1452 

1.1681 

1.1915 

FY 

2010 

2011 . 

2012 

2013 

2014 

2015 

2016 

2017 

2018 

2019 

Raw 

1.2012 

1.2252 

1 .2497 

1.2747 

1 .3002 

1:3762“ 

1.3527 

1.3798 

1.4073 

1.4355 

Weighted 

1.2153 

1.2396 

1.2644 

1.2897 

1.3155 

1.3418 

1.3687 

1.3960 

1.4240 

1.4524 
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Figure  B-l.  Schedule  Slip  Versus  Initial  Budget  Values  and  Rayleigh  Parameters 
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Figure  B-2.  Schedule  Slip  Versus  Weibull  Parameters 
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Appendix  C.  Program  Data 


Appendix  Overview 

This  appendix  provides  program  data  for  the  37  programs  included  in  the 
analysis. 
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Table  C-l.  Program  and  Cost  Growth  Data 
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Table  C-2.  Rayleigh  and  Weibull  Initial  Parameters 
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80.91 

2.48 
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1.05 
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33  FDS 

924.49 
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0.000 

34  ATARS 

202.94 

3.83 
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35  SRAMII 
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5.55 
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0.996 
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1.56 

0.000 
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Table  C-3.  Rayleigh  Final  Parameters 
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